00000800:00000200:3.1F:1734015827.106537:0:1966:0:(socklnd.c:225:ksocknal_find_peer_locked()) got peer_ni [ffff8800b34b50c0] -> 12345-192.168.201.152@tcp (2) 00000800:00000200:0.1F:1734015827.112617:0:1956:0:(socklnd.c:409:ksocknal_incr_conn_count()) Add conn type 1, ksnr_connected 2 ksnr_max_conns 1 00000800:00000200:0.1:1734015827.112629:0:1956:0:(socklnd_cb.c:748:ksocknal_queue_tx_locked()) Sending to 12345-192.168.201.152@tcp ip 192.168.201.152:988 00000800:00000200:0.1:1734015827.112632:0:1956:0:(socklnd_cb.c:767:ksocknal_queue_tx_locked()) Packet ffff8800aae4ed00 type 2, nob 96 niov 1 nkiov 0 00000800:00000200:1.1F:1734015827.115350:0:1956:0:(socklnd.c:409:ksocknal_incr_conn_count()) Add conn type 2, ksnr_connected 6 ksnr_max_conns 1 00000800:00000200:0.1:1734015827.120126:0:1956:0:(socklnd.c:409:ksocknal_incr_conn_count()) Add conn type 3, ksnr_connected e ksnr_max_conns 1 00010000:00010000:2.0:1734016141.110650:0:4728:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.110662:0:4728:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.110673:0:4728:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.110924:0:4728:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff88012dbbcd80) 00000100:00100000:2.0:1734016141.118297:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6472:x1818247334612352:12345-192.168.201.152@tcp:105: Request processed in 7722us (8539us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016141.118306:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14526 00000080:00200000:2.0:1734016141.174266:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7670335 i_blocks 64 00000020:00010000:2.0:1734016141.174277:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70deda0 (ffff8800a72d9b48/0) at cl_lock_release():233 00000020:00010000:2.0:1734016141.174281:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70deda0 (ffff8800a72d9b48/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.174288:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676e70 (ffff8800a72d9368/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.174295:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676eb0 (ffff88012db7cbd8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.174298:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70deda0 (ffff8800a72d9b48/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.174300:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676e70 (ffff8800a72d9b48/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.174307:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676eb0 (ffff8800a72d9b48/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016141.174325:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016141.174395:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016141.174405:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016141.174417:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016141.174441:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016141.174445:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2cf0, osc lock ffff8800aafce720, flags 0x400001002 00000008:00010000:2.0:1734016141.174453:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016141.174455:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2d30, osc lock ffff8800aafcf7c0, flags 0x400001002 00010000:00010000:2.0:1734016141.174475:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.174486:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.174495:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.174502:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.174515:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20b40/0x37cab3886ce5cd20 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d3178087 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.174522:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20b40/0x37cab3886ce5cd20 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d3178087 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.174529:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20b40/0x37cab3886ce5cd20 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d3178087 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.174536:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20b40/0x37cab3886ce5cd20 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d3178087 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016141.174547:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7670335 i_blocks 64 00000020:00010000:2.0:1734016141.174550:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1918 (ffff88012db7cbd8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016141.174552:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1918 (ffff88012db7cbd8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.174554:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2cf0 (ffff8800a72d9368/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.174557:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2d30 (ffff88012db7c128/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.174559:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1918 (ffff88012db7cbd8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.174561:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2cf0 (ffff88012db7cbd8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.174566:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2d30 (ffff88012db7cbd8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016141.174577:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016141.174586:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7670335 00000080:00200008:2.0:1734016141.174935:0:10366:0:(llite_lib.c:2284:ll_setattr_raw()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8) current size 7670335, valid attrs 2068, mode 8800, uid -1451891104, gid -30720, new size 7949087, atime 1734016141.000000000, mtime 1734016141.000000000, ctime 1734016141.000000000 00010000:00010000:2.0:1734016141.174964:0:10366:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5cc39 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb46129d8d3177f99 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016141.174980:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a72f1d40/0x37cab3886ce591ae lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3173bb4 expref: -99 pid: 10365 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016141.175016:0:10366:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5cc39 lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb46129d8d3177f99 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016141.175021:0:10366:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1734016141.175023:0:10366:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5cc39 lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb46129d8d3177f99 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1734016141.175065:0:10366:0:(genops.c:2298:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8800b5805800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1734016141.175074:0:10366:0:(client.c:782:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8800a73ca300 x1818247332482432/t0(0) o36->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734016141.175098:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca300 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332482432:192.168.201.152@tcp:36:fsx.0 00000100:00100000:2.0:1734016141.175178:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1734016141.183259:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334612480 00000100:00100000:2.0:1734016141.183269:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334612480 00000100:00100000:2.0:1734016141.183323:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14527 00000100:00100000:2.0:1734016141.183329:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:f1fa65c5-ac21-42fc-9fc8-6a110334833e+5:6483:x1818247334612480:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016141.183345:0:4728:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cc71 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb46129d8d3177fd8 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016141.184034:0:4728:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016141.184042:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:f1fa65c5-ac21-42fc-9fc8-6a110334833e+5:6483:x1818247334612480:12345-192.168.201.152@tcp:104: Request processed in 712us (911us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016141.184050:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14527 00000100:00100000:2.0:1734016141.194347:0:10366:0:(client.c:2968:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8800b5805800: committing for last_committed 4294969051 gen 1 00000100:00100000:2.0:1734016141.194393:0:10366:0:(client.c:3012:ptlrpc_free_committed()) @@@ stopping search req@ffff8800a73ca300 x1818247332482432/t4294969059(4294969059) o36->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 488/456 e 0 to 0 dl 1734016157 ref 3 fl Rpc:RQU/204/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000100:00100000:2.0:1734016141.194403:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ca300 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332482432:192.168.201.152@tcp:36:fsx.0 00000080:00200000:2.0:1734016141.194470:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:2.0:1734016141.194478:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:2.0:1734016141.194511:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016141.194538:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1734016141.194561:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016141.194567:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2cf0, osc lock ffff8800aafcf7c0, flags 0x0 00000008:00010000:2.0:1734016141.194571:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016141.194572:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2d30, osc lock ffff8800aafce720, flags 0x0 00010000:00010000:2.0:1734016141.194595:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.194620:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.194638:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.194647:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.194655:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.194672:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6fb4800 type 11 mode 2 res 9663677440/7 (0 18446744073709551615) 00010000:00010000:2.0:1734016141.194689:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cd27 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.194695:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cd27 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.194740:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cd27 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016141.194763:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332482688:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016141.194819:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2240 going to sleep for 11 seconds 00000100:00100000:2.0:1734016141.209845:0:4728:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016141.209867:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334612608 00000100:00100000:2.0:1734016141.209879:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334612608 00000100:00100000:2.0:1734016141.209926:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14528 00000100:00100000:2.0:1734016141.209934:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012b52fc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334612608:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016141.209954:0:4728:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20b40/0x37cab3886ce5cd20 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0xb46129d8d3178087 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.210038:0:4728:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016141.210046:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012b52fc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334612608:12345-192.168.201.152@tcp:104: Request processed in 111us (401us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016141.210053:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14528 00000100:00100000:2.0:1734016141.236976:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334612864 00000100:00100000:2.0:1734016141.236998:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334612864 00000100:00100000:2.0:1734016141.237084:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14529 00000100:00100000:2.0:1734016141.237095:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334612864:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016141.237120:0:4729:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cd27 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xb46129d8d3178095 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.237138:0:4729:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cd27 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xb46129d8d3178095 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.237150:0:4729:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cd27 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d3178095 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.237163:0:4729:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cd27 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000000000 nid: local remote: 0xb46129d8d3178095 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.237176:0:4729:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cd27 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000000000 nid: local remote: 0xb46129d8d3178095 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.237191:0:4729:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cd27 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000000 nid: local remote: 0xb46129d8d3178095 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.237240:0:4729:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff88012dbbdf80) 00000100:00100000:2.0:1734016141.237341:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334612864:12345-192.168.201.152@tcp:105: Request processed in 247us (2544us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016141.237349:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14529 00000100:00100000:2.0:1734016141.247371:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332482688:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016141.247411:0:10366:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cd27 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178095 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.247445:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cd27 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d3178095 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016141.247465:0:10366:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 3476031, setting rss=3476031; kms=3476031, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cd27 lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0xb46129d8d3178095 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.247478:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cd27 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178095 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.247485:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cd27 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178095 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016141.247631:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332483200:192.168.201.152@tcp:10:fsx.0 00000100:00100000:2.0:1734016141.247704:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016141.247710:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332483072:192.168.201.152@tcp:10:fsx.0 00000100:00100000:2.0:1734016141.254011:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332483072:192.168.201.152@tcp:10:fsx.0 00000100:00100000:2.0:1734016141.254041:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332483200:192.168.201.152@tcp:10:fsx.0 00000100:00100000:2.0:1734016141.298681:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334613376 00000100:00100000:2.0:1734016141.298698:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334613376 00000100:00100000:2.0:1734016141.298767:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14532 00000100:00100000:2.0:1734016141.298775:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334613376:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016141.298796:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.298810:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016141.298842:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016141.298844:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016141 atime: 1734016140 ctime: 1734016141 blocks: 24 00000020:00010000:2.0:1734016141.298847:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 3, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016141.298848:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016141 atime: 1734016140 ctime: 1734016141 blocks: 24 00000100:00100000:2.0:1734016141.298898:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334613376:12345-192.168.201.152@tcp:106: Request processed in 123us (1824us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016141.298905:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14532 00000100:00100000:2.0:1734016141.299640:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334613248 00000100:00100000:2.0:1734016141.299646:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334613248 00000100:00100000:2.0:1734016141.299674:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14533 00000100:00100000:2.0:1734016141.299679:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334613248:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016141.299688:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cd27 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178095 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.299700:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cd27 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178095 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016141.299714:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016141.299716:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 3754783 mtime: 1734016141 atime: 1734016140 ctime: 1734016141 blocks: 40 00000020:00010000:2.0:1734016141.299719:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016141.299719:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 3754783 mtime: 1734016141 atime: 1734016140 ctime: 1734016141 blocks: 40 00000100:00100000:2.0:1734016141.299740:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334613248:12345-192.168.201.152@tcp:106: Request processed in 60us (1094us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016141.299746:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14533 00000100:00100000:2.0:1734016141.316066:0:9532:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016141.316085:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334613504 00000100:00100000:2.0:1734016141.316096:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334613504 00000100:00100000:2.0:1734016141.316140:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334613632 00000100:00100000:2.0:1734016141.316142:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334613632 00000100:00100000:2.0:1734016141.316154:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14534 00000100:00100000:2.0:1734016141.316161:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334613504:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016141.316178:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cd27 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178095 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.316189:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cd27 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178095 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016141.316249:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016141.316251:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 3754783 mtime: 1734016141 atime: 1734016140 ctime: 1734016141 blocks: 40 00000020:00010000:2.0:1734016141.316254:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016141.316256:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 3754783 mtime: 1734016141 atime: 1734016140 ctime: 1734016141 blocks: 40 00000100:00100000:2.0:1734016141.316303:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334613504:12345-192.168.201.152@tcp:106: Request processed in 142us (419us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016141.316309:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14534 00000100:00100000:2.0:1734016141.316327:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14535 00000100:00100000:2.0:1734016141.316331:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff880137708c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334613632:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016141.316337:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.316343:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016141.316354:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016141.316355:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016141 atime: 1734016140 ctime: 1734016141 blocks: 24 00000020:00010000:2.0:1734016141.316357:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 3, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016141.316358:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016141 atime: 1734016140 ctime: 1734016141 blocks: 24 00000100:00100000:2.0:1734016141.316371:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff880137708c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334613632:12345-192.168.201.152@tcp:106: Request processed in 40us (403us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016141.316375:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14535 00000080:00200000:2.0:1734016141.320877:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7949087 i_blocks 64 00000020:00010000:2.0:1734016141.320884:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1918 (ffff88012db7cbd8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016141.320886:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1918 (ffff88012db7cbd8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.320892:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40aab0 (ffff8800a72d9368/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.320895:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40aaf0 (ffff88012db7c128/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.320898:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1918 (ffff88012db7cbd8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.320900:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40aab0 (ffff88012db7cbd8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.320903:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40aaf0 (ffff88012db7cbd8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016141.320917:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016141.320977:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 1066 00000080:00200000:2.0:1734016141.320987:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016141.320998:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016141.321010:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [1066, 1066] 00000008:00010000:2.0:1734016141.321023:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016141.321026:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8730, osc lock ffff8800aafcf560, flags 0x0 00010000:00010000:2.0:1734016141.321048:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.321056:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [172032->176127] (req 172032->176127) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.321085:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [172032->176127] (req 172032->176127) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016141.321107:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332484224:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016141.321181:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2840 going to sleep for 11 seconds 00000100:00100000:2.0:1734016141.323857:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016141.329136:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332484224:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016141.329185:0:10366:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [172032->176127] (req 172032->176127) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.329201:0:10366:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [172032->176127] (req 172032->176127) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.329211:0:10366:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [172032->176127] (req 172032->176127) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016141.331713:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000020:00010000:2.0:1734016141.364839:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ee48 (ffff88012db7cbd8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016141.364844:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ee48 (ffff88012db7cbd8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.364848:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f80b0 (ffff88012db7c128/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016141.364861:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 172032->176127) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.364872:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 172032->176127) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016141.364893:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ee48 (ffff88012db7cbd8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.364896:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f80b0 (ffff88012db7cbd8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016141.364918:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016141.364986:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 1068 00000080:00200000:2.0:1734016141.364996:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016141.365012:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016141.365027:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [1068, 1068] 00000008:00010000:2.0:1734016141.365041:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016141.365045:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8430, osc lock ffff8800aafce980, flags 0x0 00010000:00010000:2.0:1734016141.365064:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 172032->176127) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.365075:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (180224 184319) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 172032->176127) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.365085:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 172032->176127) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.365097:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 172032->176127) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.365104:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 172032->176127) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016141.375293:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ee48 (ffff88012db7cbd8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016141.375297:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ee48 (ffff88012db7cbd8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.375301:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8430 (ffff88012db7c128/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016141.375309:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 172032->176127) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.375317:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 172032->176127) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016141.375335:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ee48 (ffff88012db7cbd8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.375337:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8430 (ffff88012db7cbd8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016141.375362:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016141.375594:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016141.375611:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016141.375631:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016141.375646:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6fb6400 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:2.0:1734016141.375702:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5cd66 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016141.375710:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5cd66 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016141.375721:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5cd66 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00000100:00100000:2.0:1734016141.375747:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb480 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332485248:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016141.375988:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2600 going to sleep for 11 seconds 00000080:00200000:2.0:1734016141.392516:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7949087 i_blocks 64 00000020:00010000:2.0:1734016141.392525:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c0b68 (ffff88012db7c6c8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016141.392530:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c0b68 (ffff88012db7c6c8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.392537:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676f30 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.392542:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676f70 (ffff88012db7cea8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.392545:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c0b68 (ffff88012db7c6c8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.392547:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676f30 (ffff88012db7c6c8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.392552:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676f70 (ffff88012db7c6c8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016141.392611:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016141.392652:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016141.392660:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016141.392670:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016141.392686:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016141.392689:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e29f0, osc lock ffff8800aafce980, flags 0x400001002 00000008:00010000:2.0:1734016141.392694:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016141.392696:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2a30, osc lock ffff8800aafcfc80, flags 0x400001002 00010000:00010000:2.0:1734016141.392716:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.392729:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.392742:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.392748:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.392760:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c23180/0x37cab3886ce5cd6d lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31780e9 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.392766:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c23180/0x37cab3886ce5cd6d lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31780e9 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.392771:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c23180/0x37cab3886ce5cd6d lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d31780e9 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.392775:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c23180/0x37cab3886ce5cd6d lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31780e9 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016141.392782:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7949087 i_blocks 64 00000020:00010000:2.0:1734016141.392784:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1918 (ffff88012db7cbd8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016141.392786:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1918 (ffff88012db7cbd8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.392788:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e29f0 (ffff88012db7c128/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.392790:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2a30 (ffff8800a72d9368/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.392792:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1918 (ffff88012db7cbd8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.392793:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e29f0 (ffff88012db7cbd8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.392797:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2a30 (ffff88012db7cbd8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016141.392803:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016141.392809:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7949087 00000080:00200008:2.0:1734016141.392857:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 909815, type: SEEK_SET 00000080:00200008:2.0:1734016141.392863:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 909815, type: SEEK_SET, rc = 909815 00000080:00200008:2.0:1734016141.392871:0:10366:0:(file.c:2466:do_file_write_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 909815, count: 185 00000080:00200000:2.0:1734016141.392897:0:10366:0:(rw26.c:819:ll_write_begin()) Writing 222 of 503 to 185 bytes 00000080:00200000:2.0:1734016141.392904:0:10366:0:(file.c:2441:ll_do_tiny_write()) result: 0, original count 185 00000080:00200000:2.0:1734016141.392908:0:10366:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: write ppos: 909815, bytes: 185 00000080:00200000:2.0:1734016141.392914:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:2.0:1734016141.392921:0:10366:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b62c6a28[0x4, 72, [0x200000401:0xc:0x0] hash] 00000020:00200000:2.0:1734016141.392923:0:10366:0:(cl_io.c:195:cl_io_rw_init()) io range: 2 [909815, 910000) 0 0 00000080:00200000:2.0:1734016141.392925:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016141.392934:0:10366:0:(file.c:1893:ll_file_io_generic()) Range lock [222, 222] 00020000:00200000:2.0:1734016141.392940:0:10366:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 0 chunk: [909815, 910000) 910000, 185 00020000:00200000:2.0:1734016141.392944:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016141.392959:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [222, 222] 00000008:00010000:2.0:1734016141.392967:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016141.392969:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8430, osc lock ffff8800aafcfc80, flags 0x0 00010000:00010000:2.0:1734016141.392994:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.393002:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [909312->913407] (req 909312->913407) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.393028:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [909312->913407] (req 909312->913407) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016141.393045:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332485504:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016141.393439:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e23c0 going to sleep for 11 seconds 00000100:00100000:2.0:1734016141.403962:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016141.426757:0:4737:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016141.435013:0:4737:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1734016141.435150:0:4737:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 2/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.435167:0:4737:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1734016141.436436:0:4737:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.436455:0:4737:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cd19 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1785856->1794047) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d3178080 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.436504:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016141.436507:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016141.436513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1734016141.450009:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334614144 00000100:00100000:2.0:1734016141.450032:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334614144 00000100:00100000:2.0:1734016141.450134:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14539 00000100:00100000:2.0:1734016141.450147:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6472:x1818247334614144:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016141.450177:0:4728:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [909312->913407] (req 909312->913407) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.450190:0:4728:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [909312->913407] (req 909312->913407) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.450202:0:4728:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.450250:0:4728:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.450264:0:4728:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.450278:0:4728:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.450317:0:4728:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff88012dbbdf80) 00000100:00100000:2.0:1734016141.450404:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6472:x1818247334614144:12345-192.168.201.152@tcp:105: Request processed in 258us (820us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016141.450415:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14539 00000100:00100000:2.0:1734016141.459338:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332485760:192.168.201.152@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1734016141.464137:0:10366:0:(rw26.c:1006:ll_write_end()) pos 909815, len 185, copied 185 00000080:00200000:2.0:1734016141.464155:0:10366:0:(rw26.c:1044:ll_write_end()) page@ffff8800a71c0d80[3 ffff8800b62c6aa8 2 1 ffff8801309c1978] 00000080:00200000:2.0:1734016141.464160:0:10366:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea00049acec0 2fffff0000082d 2:0 ffff8800a71c0d80 222 lru 00000080:00200000:2.0:1734016141.464173:0:10366:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800a71c0de0 222: 1< 1 - - > 2< 909312 0 4096 0x0 0x40009 | ffff8800a97f0708 ffff88012b46a9c0 > 3< 0 0 > 4< 0 0 8 489881600 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1734016141.464176:0:10366:0:(rw26.c:1044:ll_write_end()) end page@ffff8800a71c0d80 00000080:00200000:2.0:1734016141.464177:0:10366:0:(rw26.c:1044:ll_write_end()) queued page: 1. 00000080:00200000:2.0:1734016141.464191:0:10366:0:(vvp_io.c:1218:vvp_io_write_commit()) commit async pages: 1, from 503, to 688 00000008:00010000:2.0:1734016141.464358:0:10366:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800aa7e20c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016141.464383:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 1, dirtied 1 00000080:00200000:2.0:1734016141.464389:0:10366:0:(vvp_io.c:1241:vvp_io_write_commit()) Committed 1 pages 185 bytes, tot: 185 00000080:00200000:2.0:1734016141.464396:0:10366:0:(vvp_io.c:1405:vvp_io_write_start()) f16b.sanityn: write bytes 0, result: 185 00000020:00010000:2.0:1734016141.464427:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ee48 (ffff88012db7cbd8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016141.464431:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ee48 (ffff88012db7cbd8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.464438:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8430 (ffff8800a72d9368/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016141.464452:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.464469:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016141.464499:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ee48 (ffff88012db7cbd8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.464502:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8430 (ffff88012db7cbd8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016141.464515:0:10366:0:(vvp_io.c:533:vvp_io_advance()) advancing 185 bytes 00000080:00200000:2.0:1734016141.464526:0:10366:0:(file.c:1930:ll_file_io_generic()) Range unlock [222, 222] 00000080:00200000:2.0:1734016141.464585:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016141.464589:0:10366:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 2 io complete with rc: 0, result: 185, restart: 0 00000080:00200000:2.0:1734016141.464620:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7949087 i_blocks 72 00000080:00200000:2.0:1734016141.464629:0:10366:0:(file.c:2044:ll_file_io_generic()) iot: 2, result: 185 00000080:00200008:2.0:1734016141.464695:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016141.464706:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016141.464735:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016141.464762:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012b716880/0x37cab3886ce5cd2e lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31780a3 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016141.464776:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012b716880/0x37cab3886ce5cd2e lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31780a3 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016141.464795:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016141.464800:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5cd2e 00000080:00010000:2.0:1734016141.464807:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5cd2e from it ffff8800a7343cd8 00010000:00010000:2.0:1734016141.464810:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012b716880/0x37cab3886ce5cd2e lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31780a3 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016141.464817:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012b716880/0x37cab3886ce5cd2e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31780a3 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016141.464841:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016141.464854:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016141.464869:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016141.464873:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2b70, osc lock ffff8800aafcfc80, flags 0x400001002 00000008:00010000:2.0:1734016141.464880:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016141.464881:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2bb0, osc lock ffff8800aafce980, flags 0x400001002 00010000:00010000:2.0:1734016141.464891:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.464898:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.464907:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.464914:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.464928:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 172032->176127) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.464935:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 172032->176127) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.464944:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 172032->176127) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.464951:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 172032->176127) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016141.464963:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7949087 i_blocks 72 00000020:00010000:2.0:1734016141.464966:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd240 (ffff8800a72d9518/0) at cl_lock_release():233 00000020:00010000:2.0:1734016141.464968:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd240 (ffff8800a72d9518/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.464970:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2b70 (ffff8800a72d9c68/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.464972:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2bb0 (ffff88012db7c128/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.464974:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd240 (ffff8800a72d9518/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.464976:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2b70 (ffff8800a72d9518/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.464979:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2bb0 (ffff8800a72d9518/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016141.464987:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016141.465000:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016141.465005:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016141.465009:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016141.465013:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016141.465015:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2b70, osc lock ffff8800aafce980, flags 0x400001002 00000008:00010000:2.0:1734016141.465018:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016141.465019:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2bb0, osc lock ffff8800aafcfc80, flags 0x400001002 00010000:00010000:2.0:1734016141.465024:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.465032:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.465039:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.465046:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.465056:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 172032->176127) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.465063:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 172032->176127) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.465071:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 172032->176127) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.465077:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 172032->176127) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016141.465088:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7949087 i_blocks 72 00000020:00010000:2.0:1734016141.465090:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd240 (ffff8800a72d9518/0) at cl_lock_release():233 00000020:00010000:2.0:1734016141.465092:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd240 (ffff8800a72d9518/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.465094:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2b70 (ffff88012db7c128/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.465096:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2bb0 (ffff8800a72d9c68/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.465098:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd240 (ffff8800a72d9518/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.465101:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2b70 (ffff8800a72d9518/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.465104:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2bb0 (ffff8800a72d9518/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016141.465109:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016141.465117:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7949087 00000080:00200008:2.0:1734016141.465197:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 740478, type: SEEK_SET 00000080:00200008:2.0:1734016141.465201:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 740478, type: SEEK_SET, rc = 740478 00000080:00200008:2.0:1734016141.465209:0:10366:0:(file.c:2290:do_file_read_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 740478, count: 560 00000080:00200000:2.0:1734016141.465213:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016141.465252:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016141.465290:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 180 00000080:00200000:2.0:1734016141.465299:0:10366:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: read ppos: 740478, bytes: 560 00000080:00200000:2.0:1734016141.465305:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:2.0:1734016141.465311:0:10366:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b62c6a28[0x4, 73, [0x200000401:0xc:0x0] hash] 00000020:00200000:2.0:1734016141.465314:0:10366:0:(cl_io.c:195:cl_io_rw_init()) io range: 1 [740478, 741038) 0 0 00000080:00200000:2.0:1734016141.465316:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016141.465333:0:10366:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 0 chunk: [740478, 741038) 741038, 560 00020000:00200000:2.0:1734016141.465337:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016141.465357:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [180, 180] 00000008:00010000:2.0:1734016141.465365:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016141.465367:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8430, osc lock ffff8800aafcfc80, flags 0x0 00010000:00010000:2.0:1734016141.465374:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.465381:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (737280 741375) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.465389:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 7/1,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.465395:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 7/1,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.465408:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 6/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016141.465417:0:10366:0:(vvp_io.c:840:vvp_io_read_start()) f16b.sanityn: read [740478, 741038) 00000080:00200000:2.0:1734016141.465429:0:10366:0:(vvp_io.c:899:vvp_io_read_start()) count:560 iocb pos:740478 00010000:00010000:2.0:1734016141.465463:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 5/1,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.465472:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 5/1,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.465480:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 8/2,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.465487:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 8/2,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.465493:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 7/2,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.475206:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 7/2,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.475255:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 6/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016141.559348:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016141.574413:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7949087 i_blocks 72 00000020:00010000:2.0:1734016141.574422:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd240 (ffff8800a72d9518/0) at cl_lock_release():233 00000020:00010000:2.0:1734016141.574426:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd240 (ffff8800a72d9518/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.574431:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676930 (ffff8800a72d9c68/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.574435:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676970 (ffff88012db7c488/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.574437:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd240 (ffff8800a72d9518/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.574439:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676930 (ffff8800a72d9518/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.574444:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676970 (ffff8800a72d9518/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016141.574457:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016141.574525:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016141.574538:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016141.574553:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016141.574574:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016141.574650:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2b70, osc lock ffff8800aafce980, flags 0x400001002 00000008:00010000:2.0:1734016141.574661:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016141.574662:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2bb0, osc lock ffff8800aafcfdb0, flags 0x400001002 00010000:00010000:2.0:1734016141.574689:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5cd82 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.574699:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5cd82 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.575305:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c23180/0x37cab3886ce5cd6d lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31780e9 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.575317:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c23180/0x37cab3886ce5cd6d lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31780e9 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.575329:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c23180/0x37cab3886ce5cd6d lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d31780e9 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.575338:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c23180/0x37cab3886ce5cd6d lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31780e9 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016141.578609:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016141.583234:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7949087 i_blocks 72 00000020:00010000:2.0:1734016141.583242:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1da8 (ffff88012db7c488/0) at cl_lock_release():233 00000020:00010000:2.0:1734016141.583245:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1da8 (ffff88012db7c488/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.583252:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2b70 (ffff8800a72d9c68/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.583255:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2bb0 (ffff88012db7c128/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.583259:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1da8 (ffff88012db7c488/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.583260:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2b70 (ffff88012db7c488/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.583266:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2bb0 (ffff88012db7c488/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016141.583282:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016141.583295:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7949087 00000080:00200008:2.0:1734016141.584304:0:10366:0:(llite_lib.c:2284:ll_setattr_raw()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8) current size 7949087, valid attrs 2068, mode 8800, uid -1451891104, gid -30720, new size 10207649, atime 1734016141.000000000, mtime 1734016141.000000000, ctime 1734016141.000000000 00010000:00010000:2.0:1734016141.584350:0:10366:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012b716880/0x37cab3886ce5cd2e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb46129d8d31780a3 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016141.584379:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012b715d40/0x37cab3886ce591e6 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3173c47 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016141.584446:0:10366:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012b716880/0x37cab3886ce5cd2e lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb46129d8d31780a3 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016141.584453:0:10366:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1734016141.584456:0:10366:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012b716880/0x37cab3886ce5cd2e lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb46129d8d31780a3 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1734016141.584507:0:10366:0:(genops.c:2298:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8800b3cc4800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1734016141.584523:0:10366:0:(client.c:782:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8800a73caa00 x1818247332486784/t0(0) o36->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734016141.584554:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332486784:192.168.201.152@tcp:36:fsx.0 00000100:00100000:2.0:1734016141.584700:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2b40 going to sleep for 11 seconds 00000100:00100000:2.0:1734016141.589437:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016141.594266:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332486784:192.168.201.152@tcp:36:fsx.0 00000080:00200000:2.0:1734016141.594475:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:2.0:1734016141.594481:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:2.0:1734016141.594498:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016141.594520:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1734016141.594538:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016141.594542:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2b70, osc lock ffff8800aafcfdb0, flags 0x0 00000008:00010000:2.0:1734016141.594545:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016141.594546:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2bb0, osc lock ffff8800aafce980, flags 0x0 00010000:00010000:2.0:1734016141.594571:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.594583:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.594604:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.594615:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.594623:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cd74 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 909312->913407) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31780f0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.594637:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b582d000 type 11 mode 2 res 9663677440/7 (0 18446744073709551615) 00010000:00010000:2.0:1734016141.594651:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf3c0/0x37cab3886ce5cd89 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.594661:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf3c0/0x37cab3886ce5cd89 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.594695:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf3c0/0x37cab3886ce5cd89 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016141.594714:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8700 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332487040:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016141.594773:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2240 going to sleep for 11 seconds 00000100:00100000:2.0:1734016141.600407:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14543 00000100:00100000:2.0:1734016141.600420:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334614656:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016141.600445:0:4728:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c23180/0x37cab3886ce5cd6d lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0xb46129d8d31780e9 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.600580:0:4728:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016141.600589:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334614656:12345-192.168.201.152@tcp:104: Request processed in 172us (793us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016141.600606:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14543 00000100:00100000:2.0:1734016141.600625:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14544 00000100:00100000:2.0:1734016141.600629:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334614784:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016141.600637:0:4728:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5cd5f lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 172032->176127) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d31780d4 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.600665:0:4728:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:2.0:1734016141.600670:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334614784:12345-192.168.201.152@tcp:104: Request processed in 40us (602us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016141.600675:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14544 00010000:00010000:2.0:1734016141.600702:0:4737:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1734016141.600711:0:4737:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c23180/0x37cab3886ce5cd6d lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0xb46129d8d31780e9 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.600719:0:4737:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a6c23180 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016141.600726:0:4737:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c23180/0x37cab3886ce5cd6d lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0xb46129d8d31780e9 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016141.600754:0:4737:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016141.600773:0:4737:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1734016141.600834:0:4737:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c23180/0x37cab3886ce5cd6d lrc: 2/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0xb46129d8d31780e9 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.600841:0:4737:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1734016141.600869:0:4737:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c23180/0x37cab3886ce5cd6d lrc: 1/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0xb46129d8d31780e9 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.600882:0:4737:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c23180/0x37cab3886ce5cd6d lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0xb46129d8d31780e9 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.602360:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016141.602364:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016141.602366:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1734016141.607004:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8700 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332487040:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016141.607033:0:10366:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf3c0/0x37cab3886ce5cd89 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d317810c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.607064:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf3c0/0x37cab3886ce5cd89 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d317810c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016141.607090:0:10366:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 3754783, setting rss=3754783; kms=3754783, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf3c0/0x37cab3886ce5cd89 lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0xb46129d8d317810c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.607112:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf3c0/0x37cab3886ce5cd89 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317810c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.607121:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf3c0/0x37cab3886ce5cd89 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317810c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016141.798298:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 10207649 i_blocks 104 00000080:00200000:2.0:1734016141.798321:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016141.798599:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016141.798609:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016141.798633:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016141.798657:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6fb6400 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:2.0:1734016141.798730:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5cd90 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016141.798740:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5cd90 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016141.798751:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5cd90 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00000100:00100000:2.0:1734016141.798778:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332488832:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016141.798901:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2240 going to sleep for 11 seconds 00000100:00100000:2.0:1734016141.826613:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016141.831822:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016141.863015:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016141.863167:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016141.897086:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016141.922538:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016141.930829:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016141.950845:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016141.985642:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 2629224 i_blocks 40 00000020:00010000:2.0:1734016141.985653:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ed58 (ffff88012db7c248/0) at cl_lock_release():233 00000020:00010000:2.0:1734016141.985658:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ed58 (ffff88012db7c248/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016141.985667:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676b70 (ffff88012db7c7e8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016141.985687:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cdb3 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178144 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.985727:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cdb3 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178144 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016141.985752:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676bb0 (ffff88012db7ca28/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016141.985759:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b717180/0x37cab3886ce5cdba lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317814b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.985766:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b717180/0x37cab3886ce5cdba lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317814b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016141.985779:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ed58 (ffff88012db7c248/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.985781:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676b70 (ffff88012db7c248/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016141.985790:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676bb0 (ffff88012db7c248/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016141.985859:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016141.985899:0:10366:0:(llite_lib.c:2534:ll_setattr_raw()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8) current size 2629224, valid attrs 2168, mode 8800, uid -1451891104, gid -30720, new size 2629224, atime 1734016141.000000000, mtime 1734016141.874000000, ctime 1734016141.874000000, rc 0 00000080:00200008:2.0:1734016141.985947:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016141.985956:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016141.985990:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016141.986016:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b62b8800 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:2.0:1734016141.986093:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016141.986102:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016141.986117:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00000100:00100000:2.0:1734016141.986150:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332490496:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016141.986280:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2d80 going to sleep for 11 seconds 00000100:00100000:2.0:1734016141.997719:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016141.997798:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016141.997851:0:9532:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016141.997873:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334616960 00000100:00100000:2.0:1734016141.997885:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334616960 00000100:00100000:2.0:1734016141.997943:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334617088 00000100:00100000:2.0:1734016141.997946:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334617088 00000100:00100000:2.0:1734016141.997963:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14553 00000100:00100000:2.0:1734016141.997972:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73cbb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334616960:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016141.997997:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b717180/0x37cab3886ce5cdba lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317814b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.998012:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b717180/0x37cab3886ce5cdba lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317814b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016141.998050:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016141.998053:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 0 00000020:00010000:2.0:1734016141.998056:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016141.998057:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 0 00000100:00100000:2.0:1734016141.998130:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73cbb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334616960:12345-192.168.201.152@tcp:106: Request processed in 158us (437us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016141.998139:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14553 00000100:00100000:2.0:1734016141.998171:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14554 00000100:00100000:2.0:1734016141.998177:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012b52e850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7694:x1818247334617088:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016141.998185:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cdb3 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178144 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016141.998193:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cdb3 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178144 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016141.998209:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016141.998210:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 2629224 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 40 00000020:00010000:2.0:1734016141.998214:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016141.998264:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 2629224 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 40 00000100:00100000:2.0:1734016141.998286:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012b52e850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7694:x1818247334617088:12345-192.168.201.152@tcp:106: Request processed in 108us (488us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016141.998292:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14554 00000080:00200000:2.0:1734016142.012723:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 2629224 i_blocks 40 00000020:00010000:2.0:1734016142.012734:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c2910 (ffff88012db7cab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.012737:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c2910 (ffff88012db7cab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.012745:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86764b0 (ffff88012db7c908/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.012748:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86764f0 (ffff88012db7c368/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.012751:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c2910 (ffff88012db7cab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.012753:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86764b0 (ffff88012db7cab8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.012760:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86764f0 (ffff88012db7cab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.012837:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016142.012892:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016142.012905:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016142.012919:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016142.012947:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.012950:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2630, osc lock ffff8800aafcee40, flags 0x400001002 00000008:00010000:2.0:1734016142.012959:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.012960:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2670, osc lock ffff8800aafced10, flags 0x400001002 00010000:00010000:2.0:1734016142.012982:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b582dc00 type 11 mode 6 res 10737419264/7 (0 18446744073709551615) 00010000:00010000:2.0:1734016142.012998:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5cdd6 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 4 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.013006:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5cdd6 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.013126:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cddd lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.013131:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cddd lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.020097:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016142.020178:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016142.020280:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334617344 00000100:00100000:2.0:1734016142.020294:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334617344 00000100:00100000:2.0:1734016142.020355:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334617216 00000100:00100000:2.0:1734016142.020358:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334617216 00000100:00100000:2.0:1734016142.020377:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14555 00000100:00100000:2.0:1734016142.020388:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73c8e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334617344:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016142.020414:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b717180/0x37cab3886ce5cdba lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317814b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.020431:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b717180/0x37cab3886ce5cdba lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317814b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.021231:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016142.021233:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 0 00000020:00010000:2.0:1734016142.021237:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016142.021239:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 0 00000100:00100000:2.0:1734016142.021277:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73c8e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334617344:12345-192.168.201.152@tcp:106: Request processed in 890us (1205us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.021284:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14555 00010000:00010000:2.0:1734016142.035123:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5cdd6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.035150:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5cdd6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.035170:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5cdd6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.035178:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5cdd6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.035185:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5cdd6 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.035202:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2629224, setting rss=2629224; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016142.035209:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5cdd6 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.035265:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5cdd6 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.035272:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5cdd6 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.035280:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbc000 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016142.035287:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5cdd6 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.035303:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016142.035310:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5cdd6 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.035317:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5cdd6 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.035365:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332490880:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016142.040131:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cddd lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.040172:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cddd lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.040190:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cddd lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.040200:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cddd lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.040210:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cddd lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.040299:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016142.040324:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cddd lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.040332:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cddd lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.040343:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cddd lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.040354:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbf840 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016142.040361:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cddd lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.040377:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016142.040385:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cddd lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.040392:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cddd lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.040438:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332491008:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016142.072142:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14558 00000100:00100000:2.0:1734016142.072158:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334617600:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016142.072173:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cdb3 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178144 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.072180:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cdb3 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178144 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.072241:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016142.072243:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 2629224 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 40 00000020:00010000:2.0:1734016142.072246:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016142.072247:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 2629224 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 40 00000100:00100000:2.0:1734016142.072344:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334617600:12345-192.168.201.152@tcp:106: Request processed in 192us (3699us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.072351:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14558 00000080:00200000:2.0:1734016142.083656:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 2629224 i_blocks 40 00000020:00010000:2.0:1734016142.083668:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c0490 (ffff88012db7c7e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.083672:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c0490 (ffff88012db7c7e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.083680:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676cf0 (ffff88012db7c128/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.083686:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676d30 (ffff88012db7c488/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.083690:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c0490 (ffff88012db7c7e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.083693:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676cf0 (ffff88012db7c7e8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.083701:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676d30 (ffff88012db7c7e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.083724:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016142.083777:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016142.083786:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016142.083804:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016142.083833:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.083836:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2870, osc lock ffff8800aafcf690, flags 0x400001002 00000008:00010000:2.0:1734016142.083847:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.083848:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e28b0, osc lock ffff8800aafcfa20, flags 0x400001002 00010000:00010000:2.0:1734016142.083880:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cdf9 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.083891:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cdf9 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.084011:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce00 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.084018:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce00 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.087886:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016142.087978:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016142.088032:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334617728 00000100:00100000:2.0:1734016142.088052:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334617728 00000100:00100000:2.0:1734016142.088119:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334617856 00000100:00100000:2.0:1734016142.088122:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334617856 00000100:00100000:2.0:1734016142.088140:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14559 00000100:00100000:2.0:1734016142.088151:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73cbb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334617728:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016142.088175:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b717180/0x37cab3886ce5cdba lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317814b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.088192:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b717180/0x37cab3886ce5cdba lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317814b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.088292:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016142.088295:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 0 00000020:00010000:2.0:1734016142.088300:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016142.088302:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 0 00000100:00100000:2.0:1734016142.088397:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73cbb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334617728:12345-192.168.201.152@tcp:106: Request processed in 245us (539us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.088407:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14559 00000100:00100000:2.0:1734016142.088437:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14560 00000100:00100000:2.0:1734016142.088442:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73c9880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334617856:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016142.088450:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cdb3 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178144 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.088457:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cdb3 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178144 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.088476:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016142.088478:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 2629224 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 40 00000020:00010000:2.0:1734016142.088481:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016142.088483:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 2629224 mtime: 1734016141 atime: 1734016141 ctime: 1734016141 blocks: 40 00000100:00100000:2.0:1734016142.088503:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73c9880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334617856:12345-192.168.201.152@tcp:106: Request processed in 60us (524us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.088509:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14560 00000020:00010000:2.0:1734016142.192401:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724e218 (ffff88012db7c7e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.192409:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724e218 (ffff88012db7c7e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.192417:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f80b0 (ffff88012db7c128/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016142.192429:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cdb3 lrc: 4/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178144 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.192442:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cdb3 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178144 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.192468:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724e218 (ffff88012db7c7e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.192471:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f80b0 (ffff88012db7c7e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.192485:0:10366:0:(vvp_io.c:533:vvp_io_advance()) advancing 2173 bytes 00000080:00200000:2.0:1734016142.192504:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016142.192511:0:10366:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 1 io complete with rc: 0, result: 2173, restart: 0 00000080:00200000:2.0:1734016142.192519:0:10366:0:(file.c:2044:ll_file_io_generic()) iot: 1, result: 2173 00000080:00200008:2.0:1734016142.192562:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016142.192573:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016142.192612:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016142.192637:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178159 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016142.192651:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178159 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016142.192670:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016142.192673:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5cdc1 00000080:00010000:2.0:1734016142.192680:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5cdc1 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016142.192682:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178159 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016142.192688:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178159 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016142.192712:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016142.192726:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016142.192751:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.192754:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2930, osc lock ffff8800aafcfb50, flags 0x400001002 00000008:00010000:2.0:1734016142.192763:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.192764:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2970, osc lock ffff8800aafcf8f0, flags 0x400001002 00010000:00010000:2.0:1734016142.192796:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce07 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.192805:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce07 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.193414:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5ce0e lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.193421:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5ce0e lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.193459:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332492160:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016142.214678:0:9532:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016142.214710:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334618112 00000100:00100000:2.0:1734016142.214726:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334618112 00000100:00100000:2.0:1734016142.214798:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14561 00000100:00100000:2.0:1734016142.214808:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b35a1850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7694:x1818247334618112:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016142.214836:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cdb3 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178144 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.214855:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cdb3 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178144 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.214905:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 643, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016142.214907:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 2629224 mtime: 1734016141 atime: 1734016142 ctime: 1734016141 blocks: 40 00000020:00010000:2.0:1734016142.214910:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016142.214911:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 2629224 mtime: 1734016141 atime: 1734016142 ctime: 1734016141 blocks: 40 00000100:00100000:2.0:1734016142.214986:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b35a1850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7694:x1818247334618112:12345-192.168.201.152@tcp:106: Request processed in 179us (531us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.214993:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14561 00010000:00010000:2.0:1734016142.232128:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce07 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.232140:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce07 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.232149:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce07 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.232156:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce07 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.232163:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce07 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.232178:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2629224, setting rss=2629224; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016142.232189:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce07 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.232195:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce07 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.232202:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce07 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.232209:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbd200 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016142.232212:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce07 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.232256:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016142.232260:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce07 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.232268:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce07 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.232290:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332492032:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016142.235853:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5ce1c lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.235869:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5ce1c lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.235899:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016142.235908:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332492416:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016142.249266:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:356ef100-c450-4ad1-ae36-19e8c4618593:1971:1818247332492544:192.168.201.152@tcp:400:kworker.0 00000100:00100000:2.0:1734016142.249492:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016142.249546:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334618240 00000100:00100000:2.0:1734016142.249557:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334618240 00000100:00100000:2.0:1734016142.257145:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:356ef100-c450-4ad1-ae36-19e8c4618593:1971:1818247332492544:192.168.201.152@tcp:400:kworker.0 00010000:00010000:2.0:1734016142.257162:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5ce1c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.257172:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5ce1c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.257180:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5ce1c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.257187:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5ce1c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.257193:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5ce1c lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.257206:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016142.258159:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5ce1c lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.258173:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5ce1c lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.258181:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5ce1c lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.258188:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbf600 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016142.258192:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5ce1c lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.258201:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016142.258203:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5ce1c lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.258208:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5ce1c lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.258250:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332492416:192.168.201.152@tcp:101:fsx.0 00000020:00010000:2.0:1734016142.278693:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.278699:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.278706:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b352edb0 (ffff8800a72d9c68/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016142.278722:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cdb3 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178144 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.278739:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cdb3 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178144 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.278758:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.278760:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b352edb0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.278782:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016142.278872:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016142.278886:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016142.278913:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016142.278937:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178159 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016142.278950:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178159 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016142.278972:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016142.278976:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5cdc1 00000080:00010000:2.0:1734016142.278982:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5cdc1 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016142.278986:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178159 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016142.278991:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178159 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016142.279012:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016142.279024:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016142.279048:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.279051:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c30, osc lock ffff8800aafcf300, flags 0x400001002 00000008:00010000:2.0:1734016142.279060:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.279061:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c70, osc lock ffff8800aafcf0a0, flags 0x400001002 00010000:00010000:2.0:1734016142.279090:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce23 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.279097:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce23 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.279279:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5ce2a lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.279285:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5ce2a lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.279321:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332493056:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016142.293158:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce23 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.293170:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce23 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.293177:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce23 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.293184:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce23 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.293197:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce23 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.293208:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2629224, setting rss=2629224; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016142.293273:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce23 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.293281:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce23 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.293288:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce23 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.293294:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbd200 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016142.293297:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce23 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.293306:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016142.293310:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce23 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.293316:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5ce23 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.293349:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332492928:192.168.201.152@tcp:101:fsx.0 00000080:00200000:2.0:1734016142.293439:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 2629224 i_blocks 64 00000020:00010000:2.0:1734016142.293448:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.293452:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.293461:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2c30 (ffff8800a72d9c68/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.293466:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2c70 (ffff88012db7ccf8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.293470:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.293473:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2c30 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.293480:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2c70 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.293509:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016142.293547:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016142.293555:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016142.293572:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016142.293618:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.293623:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c30, osc lock ffff8800aafcf0a0, flags 0x400001002 00000008:00010000:2.0:1734016142.293629:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.293631:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c70, osc lock ffff8800aafcf300, flags 0x400001002 00010000:00010000:2.0:1734016142.293662:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5ce31 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.293675:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5ce31 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.293776:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5ce38 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.293783:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5ce38 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.293827:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332493184:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016142.293929:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016142.293936:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332493312:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016142.312117:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5ce38 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.312141:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5ce38 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.312158:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5ce38 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.312166:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5ce38 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.312173:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5ce38 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.312192:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016142.312201:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5ce38 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.312208:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5ce38 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.312243:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5ce38 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.312253:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbd8c0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016142.312260:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5ce38 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.312276:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016142.312283:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5ce38 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.312287:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5ce38 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.315068:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332493312:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016142.315127:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5ce31 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.315143:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5ce31 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.315157:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5ce31 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.315164:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5ce31 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.315174:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5ce31 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.315188:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2629224, setting rss=2629224; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016142.315204:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5ce31 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.315209:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5ce31 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.315230:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5ce31 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.315238:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbcfc0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016142.315243:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5ce31 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.315253:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016142.315259:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5ce31 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.315267:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5ce31 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.315289:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332493184:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016142.327883:0:4729:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016142.327940:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334619008 00000100:00100000:2.0:1734016142.327956:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334619008 00000100:00100000:2.0:1734016142.328043:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14568 00000100:00100000:2.0:1734016142.328055:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff880137708c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334619008:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016142.328087:0:4729:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b717180/0x37cab3886ce5cdba lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d317814b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.328201:0:4729:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016142.328207:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff880137708c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334619008:12345-192.168.201.152@tcp:104: Request processed in 157us (1064us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.328215:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14568 00000100:00100000:2.0:1734016142.331188:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334619136 00000100:00100000:2.0:1734016142.331198:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334619136 00000100:00100000:2.0:1734016142.331286:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14569 00000100:00100000:2.0:1734016142.331298:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6472:x1818247334619136:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016142.331339:0:4728:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [479232->483327] (req 479232->483327) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.331347:0:4728:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [479232->483327] (req 479232->483327) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.331358:0:4728:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.331367:0:4728:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.331382:0:4728:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.331392:0:4728:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.331425:0:4728:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a964db00) 00000100:00100000:2.0:1734016142.331478:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6472:x1818247334619136:12345-192.168.201.152@tcp:105: Request processed in 183us (397us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.331486:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14569 00000020:00010000:2.0:1734016142.334338:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.334343:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.334348:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b35babb0 (ffff88012db7ccf8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016142.334358:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.334371:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.334386:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.334388:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b35babb0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.334407:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016142.334437:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016142.334444:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016142.334468:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016142.334488:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5cde4 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178183 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016142.334498:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5cde4 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178183 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016142.334511:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016142.334514:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5cde4 00000080:00010000:2.0:1734016142.334519:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5cde4 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016142.334522:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5cde4 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178183 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016142.334528:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5cde4 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178183 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016142.334547:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016142.334557:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016142.334574:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.334577:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2330, osc lock ffff8800aafcf300, flags 0x400001002 00000008:00010000:2.0:1734016142.334594:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.334595:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2370, osc lock ffff8800aafcf0a0, flags 0x400001002 00010000:00010000:2.0:1734016142.334609:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cdb3 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178144 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.334616:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cdb3 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178144 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.334626:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cdb3 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178144 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.334635:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cdb3 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178144 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.334661:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5ce46 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.334666:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5ce46 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.334741:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016142.334753:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332493952:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016142.337117:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334619264 00000100:00100000:2.0:1734016142.337127:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334619264 00000100:00100000:2.0:1734016142.337771:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14570 00000100:00100000:2.0:1734016142.337778:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334619264:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016142.337792:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.337802:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.337828:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016142.337830:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016141 atime: 1734016142 ctime: 1734016142 blocks: 0 00000020:00010000:2.0:1734016142.337833:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016142.337834:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016141 atime: 1734016142 ctime: 1734016142 blocks: 0 00000100:00100000:2.0:1734016142.337875:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334619264:12345-192.168.201.152@tcp:106: Request processed in 97us (887us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.337882:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14570 00010000:00010000:2.0:1734016142.341726:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5ce46 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.341749:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5ce46 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.341762:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5ce46 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.341768:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5ce46 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.341774:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5ce46 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.341788:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016142.341810:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5ce46 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.341816:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5ce46 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.341823:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5ce46 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.341828:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbc6c0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016142.341836:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5ce46 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.341849:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016142.341854:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5ce46 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.341861:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5ce46 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.341900:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332493952:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016142.346718:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334619392 00000100:00100000:2.0:1734016142.346733:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334619392 00000100:00100000:2.0:1734016142.346801:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14571 00000100:00100000:2.0:1734016142.346812:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334619392:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016142.346837:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.346851:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.346899:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016142.346902:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016141 atime: 1734016142 ctime: 1734016142 blocks: 0 00000020:00010000:2.0:1734016142.346905:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016142.346906:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016141 atime: 1734016142 ctime: 1734016142 blocks: 0 00000100:00100000:2.0:1734016142.346991:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334619392:12345-192.168.201.152@tcp:106: Request processed in 180us (453us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.347004:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14571 00000080:00200000:2.0:1734016142.350126:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8602824 i_blocks 96 00000020:00010000:2.0:1734016142.350135:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.350138:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.350144:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86760f0 (ffff88012db7c998/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.350148:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676130 (ffff88012db7ccf8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.350151:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.350153:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86760f0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.350160:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676130 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.350177:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016142.350197:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8602824 00000080:00200000:2.0:1734016142.350316:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:2.0:1734016142.350324:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:2.0:1734016142.350343:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016142.350362:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [52, 53] 00000008:00010000:2.0:1734016142.350385:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016142.350390:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8c30, osc lock ffff8800aafcf8f0, flags 0x0 00010000:00010000:2.0:1734016142.350427:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce54 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.350444:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce54 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [212992->221183] (req 212992->221183) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.350490:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce54 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [212992->221183] (req 212992->221183) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.350523:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332494208:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016142.350629:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2240 going to sleep for 11 seconds 00000100:00100000:2.0:1734016142.353917:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334619520 00000100:00100000:2.0:1734016142.353942:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334619520 00000100:00100000:2.0:1734016142.354019:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14572 00000100:00100000:2.0:1734016142.354032:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334619520:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016142.354069:0:4729:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cdb3 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d3178144 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.354200:0:4729:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016142.354210:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334619520:12345-192.168.201.152@tcp:104: Request processed in 180us (478us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.354249:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14572 00000100:00100000:2.0:1734016142.357042:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332494208:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016142.357096:0:10366:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce54 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [212992->221183] (req 212992->221183) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31781f3 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.357111:0:10366:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce54 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [212992->221183] (req 212992->221183) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31781f3 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.357122:0:10366:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce54 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [212992->221183] (req 212992->221183) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31781f3 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.363423:0:10366:0:(ldlm_request.c:185:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce54 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 212992->221183) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31781f3 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.363435:0:10366:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce54 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 212992->221183) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31781f3 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.363467:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce54 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 212992->221183) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31781f3 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.363482:0:10366:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4408520, setting rss=4408520; kms=4408520, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce54 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 212992->221183) gid 0 flags: 0x10020000020000 nid: local remote: 0xb46129d8d31781f3 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.363496:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce54 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 212992->221183) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31781f3 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.363502:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce54 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 212992->221183) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31781f3 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016142.363529:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8602824 i_blocks 96 00000100:00100000:2.0:1734016142.367334:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332494464:192.168.201.152@tcp:22:fsx.0 00000100:00100000:2.0:1734016142.378355:0:4728:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016142.378387:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334619776 00000100:00100000:2.0:1734016142.378403:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334619776 00000100:00100000:2.0:1734016142.378467:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14574 00000100:00100000:2.0:1734016142.378476:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b35a1850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7694:x1818247334619776:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016142.378494:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.378507:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.378543:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016142.378546:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 0 00000020:00010000:2.0:1734016142.378549:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016142.378552:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 0 00000100:00100000:2.0:1734016142.378618:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b35a1850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7694:x1818247334619776:12345-192.168.201.152@tcp:106: Request processed in 143us (525us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.378625:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14574 00000100:00100000:2.0:1734016142.380524:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016142.400453:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14577 00000100:00100000:2.0:1734016142.400459:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334620160:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016142.400468:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce54 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 212992->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31781f3 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.400474:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce54 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 212992->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31781f3 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.400488:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016142.400489:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4408520 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 96 00000020:00010000:2.0:1734016142.400493:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016142.400494:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4408520 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 96 00000100:00100000:2.0:1734016142.400566:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334620160:12345-192.168.201.152@tcp:106: Request processed in 105us (609us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.400574:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14577 00000100:00100000:2.0:1734016142.431031:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334620416 00000100:00100000:2.0:1734016142.431039:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334620416 00000100:00100000:2.0:1734016142.431282:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14579 00000100:00100000:2.0:1734016142.431289:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6472:x1818247334620416:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016142.431298:0:4729:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1843200->1851391] (req 1843200->1851391) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.431312:0:4729:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1843200->1851391] (req 1843200->1851391) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.431319:0:4729:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.431328:0:4729:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.431336:0:4729:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.431344:0:4729:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.431507:0:4729:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff88012b717600) 00000100:00100000:2.0:1734016142.431576:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6472:x1818247334620416:12345-192.168.201.152@tcp:105: Request processed in 286us (1237us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.431595:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14579 00000020:00010000:2.0:1734016142.435620:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.435625:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.435631:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b352ee30 (ffff88012db7ccf8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016142.435648:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.435665:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.435682:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.435686:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b352ee30 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.435711:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016142.435748:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016142.435755:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016142.435778:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016142.435793:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178159 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016142.435802:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178159 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016142.435816:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016142.435819:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5cdc1 00000080:00010000:2.0:1734016142.435824:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5cdc1 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016142.435826:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178159 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016142.435832:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178159 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016142.435851:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016142.435862:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016142.435880:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.435883:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2b70, osc lock ffff8800aafcfb50, flags 0x400001002 00000008:00010000:2.0:1734016142.435888:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.435889:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2bb0, osc lock ffff8800aafcfa20, flags 0x400001002 00010000:00010000:2.0:1734016142.435913:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce7e lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.435920:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce7e lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.435974:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.435985:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016142.435994:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332495616:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016142.436056:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.436066:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.436072:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.438523:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334620544 00000100:00100000:2.0:1734016142.438534:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334620544 00000100:00100000:2.0:1734016142.439134:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14580 00000100:00100000:2.0:1734016142.439141:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334620544:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016142.439156:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.439168:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.439200:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016142.439202:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4408520 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 96 00000020:00010000:2.0:1734016142.439205:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016142.439206:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4408520 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 96 00000100:00100000:2.0:1734016142.439799:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334620544:12345-192.168.201.152@tcp:106: Request processed in 654us (1449us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.439807:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14580 00010000:00010000:2.0:1734016142.442057:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce7e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.442071:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce7e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.442082:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce7e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.442088:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce7e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.442095:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce7e lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.442107:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4408520, setting rss=4408520; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016142.442118:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce7e lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.442128:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce7e lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.442136:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce7e lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.442142:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbd440 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016142.442146:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce7e lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.442155:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016142.442160:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce7e lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.442166:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce7e lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.442484:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332495616:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016142.450664:0:4729:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016142.450726:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334620672 00000100:00100000:2.0:1734016142.450742:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334620672 00000100:00100000:2.0:1734016142.450841:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14581 00000100:00100000:2.0:1734016142.450855:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff880137708c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334620672:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016142.450878:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.450898:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.450937:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016142.450940:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4408520 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 96 00000020:00010000:2.0:1734016142.450944:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016142.450946:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4408520 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 96 00000100:00100000:2.0:1734016142.451073:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff880137708c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334620672:12345-192.168.201.152@tcp:106: Request processed in 219us (4460us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.451084:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14581 00000100:00100000:2.0:1734016142.468334:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334620800 00000100:00100000:2.0:1734016142.468345:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334620800 00000100:00100000:2.0:1734016142.468398:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14582 00000100:00100000:2.0:1734016142.468404:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334620800:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016142.468417:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.468429:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.468452:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016142.468454:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4408520 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 96 00000020:00010000:2.0:1734016142.468457:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016142.468458:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4408520 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 96 00000100:00100000:2.0:1734016142.468505:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334620800:12345-192.168.201.152@tcp:106: Request processed in 100us (961us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.468511:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14582 00000080:00200000:2.0:1734016142.473512:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8602824 i_blocks 96 00000020:00010000:2.0:1734016142.473520:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.473524:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.473530:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676f30 (ffff88012db7ccf8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.473533:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676f70 (ffff88012db7c998/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.473538:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.473540:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676f30 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.473546:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676f70 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.473561:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016142.473596:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016142.473602:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016142.473611:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016142.473626:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.473629:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2270, osc lock ffff8800aafcf690, flags 0x400001002 00000008:00010000:2.0:1734016142.473635:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.473636:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e22b0, osc lock ffff8800aafced10, flags 0x400001002 00010000:00010000:2.0:1734016142.473663:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce93 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.473670:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce93 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.473730:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.473749:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332496000:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016142.473821:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.473832:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.473839:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.483027:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334620928 00000100:00100000:2.0:1734016142.483052:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334620928 00000100:00100000:2.0:1734016142.483144:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14583 00000100:00100000:2.0:1734016142.483156:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334620928:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016142.483182:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.483199:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.483311:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016142.483313:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4408520 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 96 00000020:00010000:2.0:1734016142.483317:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016142.483318:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4408520 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 96 00000100:00100000:2.0:1734016142.483427:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334620928:12345-192.168.201.152@tcp:106: Request processed in 275us (544us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.483435:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14583 00010000:00010000:2.0:1734016142.487619:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce93 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.487641:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce93 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.487656:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce93 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.487663:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce93 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.487671:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce93 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.487693:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4408520, setting rss=4408520; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016142.487708:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce93 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.487715:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce93 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.487721:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce93 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.487728:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbd440 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016142.487735:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce93 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.487756:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016142.487764:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce93 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.487769:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5ce93 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.487824:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332496000:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016142.498037:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334621056 00000100:00100000:2.0:1734016142.498052:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334621056 00000100:00100000:2.0:1734016142.498117:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14584 00000100:00100000:2.0:1734016142.498125:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334621056:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016142.498141:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.498156:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.498201:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016142.498205:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 2874777 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 1 00000020:00010000:2.0:1734016142.498208:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 3, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016142.498209:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 2874777 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 1 00000100:00100000:2.0:1734016142.498295:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334621056:12345-192.168.201.152@tcp:106: Request processed in 168us (1903us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.498305:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14584 00000080:00200000:2.0:1734016142.501212:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8602824 i_blocks 97 00000020:00010000:2.0:1734016142.501242:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.501245:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.501249:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86764b0 (ffff88012db7ccf8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.501253:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86764f0 (ffff88012db7c998/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.501255:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.501257:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86764b0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.501261:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86764f0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.501272:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016142.501293:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016142.501299:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016142.501309:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016142.501326:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.501329:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2b70, osc lock ffff8800aafcee40, flags 0x400001002 00000008:00010000:2.0:1734016142.501334:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.501335:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2bb0, osc lock ffff8800aafce980, flags 0x400001002 00010000:00010000:2.0:1734016142.501351:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.501363:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.501373:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.501381:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.501413:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5cea1 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.501418:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5cea1 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.501473:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016142.501485:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332496256:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016142.503976:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334621184 00000100:00100000:2.0:1734016142.503985:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334621184 00000100:00100000:2.0:1734016142.504036:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14585 00000100:00100000:2.0:1734016142.504045:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334621184:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016142.504061:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.504070:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.504098:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016142.504100:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 2874777 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 1 00000020:00010000:2.0:1734016142.504103:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 3, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016142.504104:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 2874777 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 1 00000100:00100000:2.0:1734016142.504149:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334621184:12345-192.168.201.152@tcp:106: Request processed in 106us (282us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.504156:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14585 00010000:00010000:2.0:1734016142.510862:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5cea1 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.510882:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5cea1 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.510895:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5cea1 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.510902:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5cea1 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.510908:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5cea1 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.510919:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2874777, setting rss=2874777; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016142.510933:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5cea1 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.510938:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5cea1 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.510944:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5cea1 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.510951:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbc6c0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016142.510956:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5cea1 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.510966:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016142.510971:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5cea1 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.510976:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5cea1 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.511121:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332496256:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016142.514133:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334621312 00000100:00100000:2.0:1734016142.514152:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334621312 00000020:00010000:2.0:1734016142.564740:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.564745:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.564750:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f87b0 (ffff88012db7c998/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016142.564760:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.564774:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.564797:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.564799:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f87b0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.564826:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016142.564845:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016142.564860:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016142.564877:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [558, 558] 00000008:00010000:2.0:1734016142.564897:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016142.564905:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8db0, osc lock ffff8800aafcfdb0, flags 0x0 00010000:00010000:2.0:1734016142.564930:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.564945:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2285568 2289663) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.564957:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.564965:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.564970:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.565011:0:10366:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016142.565026:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af698248, count 1, dirtied 1 00000020:00010000:2.0:1734016142.565040:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.565042:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.565044:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8db0 (ffff88012db7c998/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016142.565047:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.565053:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.565067:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.565069:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8db0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.565080:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016142.565154:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 559 00000080:00200000:2.0:1734016142.565160:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016142.565164:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016142.565169:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [559, 559] 00000008:00010000:2.0:1734016142.565173:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016142.565176:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8db0, osc lock ffff8800aafcfdb0, flags 0x0 00010000:00010000:2.0:1734016142.565183:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.565191:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2289664 2293759) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.565198:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 8/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.565204:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 8/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.565208:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 7/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.568309:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00010000:2.0:1734016142.568330:0:1971:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e26c0 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.568380:0:1971:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2289664 2293759) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016142.568433:0:1971:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73cb480 - object 0x280000400:7 offset 2289664<>2293760 00000100:00100000:2.0:1734016142.568466:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332496896:192.168.201.152@tcp:3:fsx.0 00000020:00010000:2.0:1734016142.572829:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.572835:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.572839:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8db0 (ffff88012db7c998/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016142.572850:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.572862:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.572881:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.572883:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8db0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.572910:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016142.572929:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016142.572944:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016142.572956:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [559, 559] 00000008:00010000:2.0:1734016142.572967:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016142.572972:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8db0, osc lock ffff8800aafcfdb0, flags 0x0 00010000:00010000:2.0:1734016142.572991:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.573003:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2289664 2293759) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.573014:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.573021:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.573027:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.573065:0:10366:0:(osc_cache.c:543:osc_extent_merge()) ### extent: ffff8800aa7e2300 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.573072:0:10366:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2300 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.573078:0:10366:0:(osc_cache.c:560:osc_extent_merge()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.573087:0:10366:0:(osc_cache.c:771:osc_extent_find()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016142.573101:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af698248, count 1, dirtied 1 00000020:00010000:2.0:1734016142.573112:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.573115:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.573117:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8db0 (ffff88012db7cf38/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016142.573121:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.573129:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.573140:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.573141:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8db0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.573148:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016142.573196:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 560 00000080:00200000:2.0:1734016142.573201:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016142.573206:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016142.573210:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [560, 560] 00000008:00010000:2.0:1734016142.573215:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016142.574558:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8db0, osc lock ffff8800aafcfdb0, flags 0x0 00010000:00010000:2.0:1734016142.574597:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.574612:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2293760 2297855) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.574620:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.574629:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.574636:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.574820:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00010000:2.0:1734016142.574842:0:1971:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e2300 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.574874:0:1971:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2293760 2297855) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016142.574929:0:1971:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73cbb80 - object 0x280000400:7 offset 2293760<>2297856 00000100:00100000:2.0:1734016142.574961:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332497152:192.168.201.152@tcp:3:fsx.0 00000020:00010000:2.0:1734016142.583100:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.583105:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.583109:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8db0 (ffff88012db7cf38/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016142.583118:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.583130:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.583150:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.583152:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8db0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.583177:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016142.583194:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016142.583209:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016142.583237:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [560, 560] 00000008:00010000:2.0:1734016142.583249:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016142.583254:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8db0, osc lock ffff8800aafcfdb0, flags 0x0 00010000:00010000:2.0:1734016142.583277:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.583290:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2293760 2297855) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.583300:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.583305:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.583311:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.583340:0:10366:0:(osc_cache.c:543:osc_extent_merge()) ### extent: ffff8800aa7e2cc0 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.583346:0:10366:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2cc0 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.583352:0:10366:0:(osc_cache.c:560:osc_extent_merge()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.583358:0:10366:0:(osc_cache.c:771:osc_extent_find()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016142.583371:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af698248, count 1, dirtied 1 00000020:00010000:2.0:1734016142.583379:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.583381:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.583383:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8db0 (ffff88012db7c998/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016142.583385:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.583390:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.583396:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.583397:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8db0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.583402:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016142.583428:0:10366:0:(file.c:5185:ll_fsync()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8), start 2285568, end 2297855, datasync 1 00000080:00200000:2.0:1734016142.583433:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016142.583437:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1734016142.583444:0:10366:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.583454:0:10366:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.583604:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2285568 2289663) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016142.583665:0:10366:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73c9880 - object 0x280000400:7 offset 2285568<>2297856 00000080:00200000:2.0:1734016142.592575:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1734016142.592665:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332497408:192.168.201.152@tcp:44:fsx.0 00000100:00100000:2.0:1734016142.592745:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e29c0 going to sleep for 11 seconds 00000100:00100000:2.0:1734016142.600697:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332497408:192.168.201.152@tcp:44:fsx.0 00000080:00200000:2.0:1734016142.600760:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016142.600785:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1734016142.609979:0:1971:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8800b5805800: committing for last_committed 4294969892 gen 2 00000100:00100000:2.0:1734016142.610106:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332497536:192.168.201.152@tcp:16:fsx.0 00000100:00100000:2.0:1734016142.621459:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016142.629321:0:9532:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016142.629343:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334621696 00000100:00100000:2.0:1734016142.629351:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334621696 00000100:00100000:2.0:1734016142.629407:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14588 00000100:00100000:2.0:1734016142.629415:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b35a1850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334621696:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016142.629428:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.629438:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.629463:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 386, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016142.629465:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4408520 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 120 00000020:00010000:2.0:1734016142.629468:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016142.629469:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4408520 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 120 00000100:00100000:2.0:1734016142.629517:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b35a1850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334621696:12345-192.168.201.152@tcp:106: Request processed in 103us (964us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.629525:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14588 00000080:00200000:2.0:1734016142.637780:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8602824 i_blocks 120 00000020:00010000:2.0:1734016142.637794:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.637799:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.637810:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40aab0 (ffff88012db7ccf8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.637817:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40aaf0 (ffff88012db7c998/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.637822:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.637825:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40aab0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.637831:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40aaf0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.637850:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016142.637869:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8602824 00000080:00200000:2.0:1734016142.637970:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:2.0:1734016142.637980:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:2.0:1734016142.638006:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016142.638026:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1094, 1094] 00000008:00010000:2.0:1734016142.638052:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016142.638058:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8a30, osc lock ffff8800aafcfdb0, flags 0x0 00010000:00010000:2.0:1734016142.638098:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.638107:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [286720->290815] (req 286720->290815) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.638149:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [286720->290815] (req 286720->290815) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.638191:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332497920:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016142.638304:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2c00 going to sleep for 11 seconds 00000100:00100000:2.0:1734016142.648981:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334621824 00000100:00100000:2.0:1734016142.648990:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334621824 00000100:00100000:2.0:1734016142.649029:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14589 00000100:00100000:2.0:1734016142.649038:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341d880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334621824:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016142.649055:0:4729:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.649135:0:4729:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016142.649141:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341d880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334621824:12345-192.168.201.152@tcp:104: Request processed in 103us (271us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.649148:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14589 00000100:00100000:2.0:1734016142.659307:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332497920:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016142.659351:0:10366:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 4/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [286720->290815] (req 286720->290815) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.659364:0:10366:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 4/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [286720->290815] (req 286720->290815) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.659373:0:10366:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 4/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [286720->290815] (req 286720->290815) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016142.659626:0:4736:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016142.659692:0:4736:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1734016142.659730:0:4736:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 2/0,0 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.659738:0:4736:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1734016142.660175:0:4736:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 1/0,0 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.660187:0:4736:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5ce3f lrc: 0/0,0 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 479232->483327) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d31781de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.660212:0:4736:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016142.660214:0:4736:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016142.664561:0:4736:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1734016142.664596:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016142.664625:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332498176:192.168.201.152@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1734016142.673827:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334621952 00000100:00100000:2.0:1734016142.673843:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334621952 00000100:00100000:2.0:1734016142.673897:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14590 00000100:00100000:2.0:1734016142.673910:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6472:x1818247334621952:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016142.673932:0:9532:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [286720->290815] (req 286720->290815) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.673944:0:9532:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [286720->290815] (req 286720->290815) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.673956:0:9532:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 286720->290815) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.673970:0:9532:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 286720->290815) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.673982:0:9532:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 286720->290815) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.673998:0:9532:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 286720->290815) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.674040:0:9532:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff88012dbbcfc0) 00000100:00100000:2.0:1734016142.674125:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6472:x1818247334621952:12345-192.168.201.152@tcp:105: Request processed in 217us (433us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.674135:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14590 00000100:00100000:2.0:1734016142.681508:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332498176:192.168.201.152@tcp:103:ldlm_bl.0 00000020:00010000:2.0:1734016142.687139:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.687143:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.687148:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8a30 (ffff88012db7c998/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016142.687160:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 286720->290815) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.687173:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 286720->290815) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.687190:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.687193:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8a30 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.687211:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016142.687272:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016142.687280:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016142.687302:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016142.687320:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178159 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016142.687331:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178159 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016142.687346:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016142.687349:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5cdc1 00000080:00010000:2.0:1734016142.687353:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5cdc1 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016142.687356:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178159 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016142.687361:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178159 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016142.687382:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016142.687396:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016142.687410:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.687412:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2630, osc lock ffff8800aafcfdb0, flags 0x400001002 00000008:00010000:2.0:1734016142.687417:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.687419:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2670, osc lock ffff8800aafcfc80, flags 0x400001002 00010000:00010000:2.0:1734016142.687446:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5cec4 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.687459:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5cec4 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.687523:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cecb lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.687528:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cecb lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.687565:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332498560:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016142.687640:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016142.687645:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332498432:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016142.691600:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cecb lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.691617:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cecb lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.691628:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cecb lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.691634:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cecb lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.691643:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cecb lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.691661:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2874777, setting rss=2874777; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016142.691670:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cecb lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.691677:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cecb lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.691689:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cecb lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.691699:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbd200 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016142.691704:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cecb lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.691715:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016142.691720:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cecb lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.691726:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cecb lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.693337:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332498560:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016142.695380:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5cec4 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.695393:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5cec4 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.695405:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5cec4 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.695411:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5cec4 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.695419:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5cec4 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.695431:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4408520, setting rss=4408520; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016142.695451:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5cec4 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.695457:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5cec4 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.695463:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5cec4 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.695469:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbd8c0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016142.699444:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5cec4 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.699461:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016142.699470:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5cec4 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.699478:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5cec4 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.699504:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332498432:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016142.701188:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334622336 00000100:00100000:2.0:1734016142.701201:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334622336 00000100:00100000:2.0:1734016142.701284:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14593 00000100:00100000:2.0:1734016142.701294:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334622336:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016142.701311:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 286720->290815) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.701324:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 286720->290815) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.701364:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 386, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016142.701367:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 2874777 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 16 00000020:00010000:2.0:1734016142.701370:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016142.701371:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 2874777 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 16 00000100:00100000:2.0:1734016142.701439:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334622336:12345-192.168.201.152@tcp:106: Request processed in 147us (369us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.701446:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14593 00000100:00100000:2.0:1734016142.701716:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016142.701738:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334622464 00000100:00100000:2.0:1734016142.701741:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334622464 00000100:00100000:2.0:1734016142.701759:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14594 00000100:00100000:2.0:1734016142.701763:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73c9880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334622464:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016142.701772:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.701778:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.701794:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 386, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016142.701797:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4408520 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 120 00000020:00010000:2.0:1734016142.701799:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016142.701800:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4408520 mtime: 1734016142 atime: 1734016142 ctime: 1734016142 blocks: 120 00000100:00100000:2.0:1734016142.701817:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73c9880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334622464:12345-192.168.201.152@tcp:106: Request processed in 54us (101us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.701822:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14594 00000080:00200000:2.0:1734016142.710674:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8602824 i_blocks 136 00000020:00010000:2.0:1734016142.710681:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.710684:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.710691:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676630 (ffff88012db7c128/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.710696:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676670 (ffff88012db7c998/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.710698:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.710700:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676630 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.710706:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676670 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.710723:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016142.710739:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8602824 00000080:00200008:2.0:1734016142.710769:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016142.710777:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016142.710810:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016142.710833:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5cde4 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178183 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016142.710895:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5cde4 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178183 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016142.710918:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016142.710921:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5cde4 00000080:00010000:2.0:1734016142.710929:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5cde4 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016142.710933:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5cde4 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178183 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016142.710940:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5cde4 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178183 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016142.710970:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016142.710987:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016142.711015:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.711019:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2930, osc lock ffff8800aafcf560, flags 0x400001002 00000008:00010000:2.0:1734016142.711026:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.711028:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2970, osc lock ffff8800aafce720, flags 0x400001002 00010000:00010000:2.0:1734016142.711045:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.711053:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.711062:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.711068:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.711092:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 286720->290815) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.711100:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 286720->290815) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.711106:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 286720->290815) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.711112:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 286720->290815) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016142.711129:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8602824 i_blocks 136 00000020:00010000:2.0:1734016142.711132:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.711134:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.711137:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2930 (ffff88012db7c998/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.711139:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2970 (ffff88012db7c128/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.711141:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.711142:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2930 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.711148:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2970 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.711155:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016142.711173:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016142.711177:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016142.711180:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016142.711184:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.711185:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2930, osc lock ffff8800aafce720, flags 0x400001002 00000008:00010000:2.0:1734016142.711190:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.711191:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2970, osc lock ffff8800aafcf560, flags 0x400001002 00010000:00010000:2.0:1734016142.711197:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.711205:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.711214:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.711249:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.711259:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 286720->290815) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.711266:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 286720->290815) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.711272:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 286720->290815) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.711277:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 286720->290815) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016142.711285:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8602824 i_blocks 136 00000020:00010000:2.0:1734016142.711287:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.711289:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.711291:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2930 (ffff88012db7c128/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.711293:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2970 (ffff88012db7c998/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.711295:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.711297:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2930 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.711300:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2970 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.711303:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016142.711308:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8602824 00000080:00200008:2.0:1734016142.711401:0:10366:0:(llite_lib.c:2284:ll_setattr_raw()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8) current size 8602824, valid attrs 2068, mode 8800, uid -1451891104, gid -30720, new size 6800353, atime 1734016142.000000000, mtime 1734016142.000000000, ctime 1734016142.000000000 00010000:00010000:2.0:1734016142.711438:0:10366:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb46129d8d3178159 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016142.711457:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012b715d40/0x37cab3886ce591e6 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3173c47 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016142.711508:0:10366:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb46129d8d3178159 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016142.711514:0:10366:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1734016142.711516:0:10366:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5cdc1 lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb46129d8d3178159 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1734016142.711567:0:10366:0:(genops.c:2298:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8800b3cc4800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1734016142.711585:0:10366:0:(client.c:782:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8800a73cb480 x1818247332499072/t0(0) o36->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734016142.711615:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb480 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332499072:192.168.201.152@tcp:36:fsx.0 00000100:00100000:2.0:1734016142.711744:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2900 going to sleep for 11 seconds 00000100:00100000:2.0:1734016142.719875:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016142.741329:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334622720 00000100:00100000:2.0:1734016142.741354:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334622720 00000100:00100000:2.0:1734016142.741490:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14596 00000100:00100000:2.0:1734016142.741499:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341d880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334622720:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016142.741520:0:9532:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b717600/0x37cab3886ce5ce77 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1843200->1851391) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d3178216 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.742164:0:9532:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016142.742174:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341d880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334622720:12345-192.168.201.152@tcp:104: Request processed in 674us (5496us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.742183:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14596 00000100:00100000:2.0:1734016142.749398:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334622848 00000100:00100000:2.0:1734016142.749413:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334622848 00000100:00100000:2.0:1734016142.749490:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14597 00000100:00100000:2.0:1734016142.749501:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6470:x1818247334622848:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016142.749518:0:9532:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23600/0x37cab3886ce5cee0 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xb46129d8d3178286 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.749533:0:9532:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23600/0x37cab3886ce5cee0 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xb46129d8d3178286 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.749544:0:9532:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23600/0x37cab3886ce5cee0 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d3178286 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.749561:0:9532:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23600/0x37cab3886ce5cee0 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000000000 nid: local remote: 0xb46129d8d3178286 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.749591:0:9532:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23600/0x37cab3886ce5cee0 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000000000 nid: local remote: 0xb46129d8d3178286 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.749602:0:9532:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23600/0x37cab3886ce5cee0 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000000000 nid: local remote: 0xb46129d8d3178286 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.749630:0:9532:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a6c23600) 00000100:00100000:2.0:1734016142.749729:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6470:x1818247334622848:12345-192.168.201.152@tcp:105: Request processed in 230us (477us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.749737:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14597 00000100:00100000:2.0:1734016142.756382:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334622976 00000100:00100000:2.0:1734016142.756396:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334622976 00000100:00100000:2.0:1734016142.756770:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14598 00000100:00100000:2.0:1734016142.756778:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334622976:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016142.756800:0:4728:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5cebd lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 286720->290815) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d317825c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.757425:0:4728:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016142.757434:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334622976:12345-192.168.201.152@tcp:104: Request processed in 656us (1915us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.757442:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14598 00000100:00100000:2.0:1734016142.766690:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334623104 00000100:00100000:2.0:1734016142.766697:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334623104 00000100:00100000:2.0:1734016142.766740:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14599 00000100:00100000:2.0:1734016142.766748:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6470:x1818247334623104:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016142.766760:0:4728:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.766769:0:4728:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.766778:0:4728:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.766786:0:4728:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.766794:0:4728:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.766801:0:4728:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.766819:0:4728:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a6c22d00) 00000100:00100000:2.0:1734016142.766861:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6470:x1818247334623104:12345-192.168.201.152@tcp:105: Request processed in 113us (538us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.766867:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14599 00000100:00100000:2.0:1734016142.842332:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334623744 00000100:00100000:2.0:1734016142.842352:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334623744 00000100:00100000:2.0:1734016142.842588:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14604 00000100:00100000:2.0:1734016142.842603:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334623744:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016142.842634:0:9532:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23600/0x37cab3886ce5cee0 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d3178286 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.843068:0:9532:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016142.843079:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334623744:12345-192.168.201.152@tcp:104: Request processed in 479us (998us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.843091:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14604 00000100:00100000:2.0:1734016142.851303:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334623872 00000100:00100000:2.0:1734016142.851318:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334623872 00000100:00100000:2.0:1734016142.851404:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14605 00000100:00100000:2.0:1734016142.851415:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6470:x1818247334623872:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016142.851432:0:9532:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21200/0x37cab3886ce5cf11 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [3858432->3866623] (req 3858432->3866623) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31782be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.851446:0:9532:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21200/0x37cab3886ce5cf11 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [3858432->3866623] (req 3858432->3866623) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31782be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.851459:0:9532:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21200/0x37cab3886ce5cf11 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3858432->3866623) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31782be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.851469:0:9532:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21200/0x37cab3886ce5cf11 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3858432->3866623) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31782be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.851478:0:9532:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21200/0x37cab3886ce5cf11 lrc: 6/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3858432->3866623) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31782be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.851493:0:9532:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21200/0x37cab3886ce5cf11 lrc: 6/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3858432->3866623) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d31782be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.851536:0:9532:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a6c21200) 00000100:00100000:2.0:1734016142.851642:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6470:x1818247334623872:12345-192.168.201.152@tcp:105: Request processed in 228us (2110us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.851649:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14605 00000020:00010000:2.0:1734016142.946962:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724e038 (ffff88012db7c488/0) at cl_lock_release():233 00000020:00010000:2.0:1734016142.946966:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724e038 (ffff88012db7c488/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016142.946969:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f8330 (ffff88012db7cc68/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016142.946977:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21200/0x37cab3886ce5cf11 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3858432->3866623) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31782be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.946987:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21200/0x37cab3886ce5cf11 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3858432->3866623) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31782be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016142.947001:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724e038 (ffff88012db7c488/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016142.947003:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f8330 (ffff88012db7c488/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016142.947011:0:10366:0:(vvp_io.c:533:vvp_io_advance()) advancing 2676 bytes 00000080:00200000:2.0:1734016142.947020:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016142.947023:0:10366:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 1 io complete with rc: 0, result: 2676, restart: 0 00000080:00200000:2.0:1734016142.947028:0:10366:0:(file.c:2044:ll_file_io_generic()) iot: 1, result: 2676 00000080:00200008:2.0:1734016142.947071:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016142.947078:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016142.947099:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016142.947113:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b62b8800 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:2.0:1734016142.947160:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016142.947171:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016142.947181:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00000100:00100000:2.0:1734016142.947207:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9180 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332501248:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016142.952467:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e23c0 going to sleep for 11 seconds 00000100:00100000:2.0:1734016142.952567:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9180 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332501248:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016142.952608:0:10366:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d31782cc expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016142.952619:0:10366:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d31782cc expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016142.952636:0:10366:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d31782cc expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016142.952650:0:10366:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d31782cc expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016142.952665:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d31782cc expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016142.952673:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d31782cc expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016142.952681:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d31782cc expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1734016142.952702:0:10366:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a73c9180 x1818247332501248/t0(0) o101->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016158 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:2.0:1734016142.952716:0:10366:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d31782cc expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1734016142.952734:0:10366:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d31782cc expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016142.952753:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b62b8800 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:2.0:1734016142.952852:0:10366:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1734016142.952860:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5cf18 00000080:00010000:2.0:1734016142.952868:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016142.952871:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5cf18 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016142.952875:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31782cc expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016142.952885:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31782cc expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016142.952943:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016142.952960:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016142.952984:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.952986:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e23f0, osc lock ffff8800aafcf560, flags 0x400001002 00000008:00010000:2.0:1734016142.952995:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016142.952996:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2430, osc lock ffff8800aafce720, flags 0x400001002 00010000:00010000:2.0:1734016142.953030:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cf1f lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.953044:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cf1f lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.953129:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.953150:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016142.953171:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332501376:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016142.961205:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cf1f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.961566:0:1970:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cf1f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xb46129d8d31782d3 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.961595:0:1970:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cf1f lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d31782d3 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.961612:0:1970:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cf1f lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xb46129d8d31782d3 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016142.961634:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cf1f lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31782d3 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.962180:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cf1f lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d31782d3 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.962193:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cf1f lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d31782d3 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.962200:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cf1f lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d31782d3 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.962205:0:1970:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5cf1f lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31782d3 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.962238:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332501376:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016142.972433:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14607 00000100:00100000:2.0:1734016142.972442:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7694:x1818247334624128:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016142.972455:0:4728:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c21200/0x37cab3886ce5cf11 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3858432->3866623) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d31782be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.974578:0:4728:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:2.0:1734016142.974610:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7694:x1818247334624128:12345-192.168.201.152@tcp:104: Request processed in 2153us (2732us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016142.974630:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14607 00000100:00100000:2.0:1734016142.995361:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341c700 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332501504:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016142.995408:0:10366:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964e640/0x37cab3886ce5cf26 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [4616192->4624383] (req 4616192->4624383) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31782da expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.995424:0:10366:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964e640/0x37cab3886ce5cf26 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [4616192->4624383] (req 4616192->4624383) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31782da expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016142.995436:0:10366:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964e640/0x37cab3886ce5cf26 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [4616192->4624383] (req 4616192->4624383) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31782da expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016142.999613:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016143.012702:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016143.024512:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016143.028047:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8815949 i_blocks 97 00000020:00010000:2.0:1734016143.028061:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c3908 (ffff88012db7cc68/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.028067:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c3908 (ffff88012db7cc68/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.028077:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86764b0 (ffff88012db7cd88/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.028084:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86764f0 (ffff88012db7c1b8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.028087:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c3908 (ffff88012db7cc68/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.028089:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86764b0 (ffff88012db7cc68/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.028099:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86764f0 (ffff88012db7cc68/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.028117:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016143.028134:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8815949 00000080:00200008:2.0:1734016143.028290:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 5518357, type: SEEK_SET 00000080:00200008:2.0:1734016143.028302:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 5518357, type: SEEK_SET, rc = 5518357 00000080:00200008:2.0:1734016143.028323:0:10366:0:(file.c:2290:do_file_read_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 5518357, count: 5372 00000080:00200000:2.0:1734016143.028333:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016143.028349:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016143.028424:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 1347 00000080:00200000:2.0:1734016143.028444:0:10366:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: read ppos: 5518357, bytes: 5372 00000080:00200000:2.0:1734016143.028451:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:2.0:1734016143.028460:0:10366:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b62c6a28[0x4, 2, [0x200000401:0xc:0x0] hash] 00000020:00200000:2.0:1734016143.028463:0:10366:0:(cl_io.c:195:cl_io_rw_init()) io range: 1 [5518357, 5523729) 0 0 00000080:00200000:2.0:1734016143.028467:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016143.028485:0:10366:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 1 chunk: [5518357, 5523729) 5523729, 5372 00020000:00200000:2.0:1734016143.028490:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016143.029269:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [1347, 1348] 00000008:00010000:2.0:1734016143.029290:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016143.029295:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8a30, osc lock ffff8800aafcf7c0, flags 0x0 00010000:00010000:2.0:1734016143.029323:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.029337:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1323008 1331199) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.029349:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 7/1,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.029356:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 7/1,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.029362:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 6/1,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016143.029369:0:10366:0:(vvp_io.c:840:vvp_io_read_start()) f16b.sanityn: read [5518357, 5523729) 00000080:00200000:2.0:1734016143.029386:0:10366:0:(vvp_io.c:899:vvp_io_read_start()) count:5372 iocb pos:5518357 00000020:00010000:2.0:1734016143.042314:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724e308 (ffff88012db7cc68/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.042320:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724e308 (ffff88012db7cc68/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.042325:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8a30 (ffff88012db7c1b8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.042335:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/1,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.042352:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.042368:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724e308 (ffff88012db7cc68/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.042370:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8a30 (ffff88012db7cc68/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.042384:0:10366:0:(vvp_io.c:533:vvp_io_advance()) advancing 5372 bytes 00000080:00200000:2.0:1734016143.042410:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016143.042415:0:10366:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 1 io complete with rc: 0, result: 5372, restart: 0 00000080:00200000:2.0:1734016143.042422:0:10366:0:(file.c:2044:ll_file_io_generic()) iot: 1, result: 5372 00000080:00200008:2.0:1734016143.042466:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016143.042476:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016143.042504:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016143.042532:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31782cc expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016143.042545:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31782cc expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016143.042558:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016143.042561:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5cf18 00000080:00010000:2.0:1734016143.042568:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5cf18 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016143.042582:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31782cc expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016143.042588:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31782cc expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016143.042618:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016143.042632:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016143.042647:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.042649:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c30, osc lock ffff8800aafcf7c0, flags 0x400001002 00000008:00010000:2.0:1734016143.042654:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.042656:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c70, osc lock ffff8800aafce5f0, flags 0x400001002 00010000:00010000:2.0:1734016143.042676:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5cf3b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.042683:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5cf3b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.042766:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.042773:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.042783:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.042790:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.045492:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016143.052317:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8815949 i_blocks 96 00000020:00010000:2.0:1734016143.052324:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c3908 (ffff88012db7cc68/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.052327:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c3908 (ffff88012db7cc68/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.052332:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2c30 (ffff88012db7cd88/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.052337:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2c70 (ffff88012db7c1b8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.052340:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c3908 (ffff88012db7cc68/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.052341:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2c30 (ffff88012db7cc68/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.052347:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2c70 (ffff88012db7cc68/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.052362:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016143.052391:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016143.052397:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016143.052409:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016143.052428:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.052431:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c30, osc lock ffff8800aafce5f0, flags 0x400001002 00000008:00010000:2.0:1734016143.052438:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.052439:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c70, osc lock ffff8800aafcf7c0, flags 0x400001002 00010000:00010000:2.0:1734016143.052466:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5cf42 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.052475:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5cf42 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.052583:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.052592:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.052601:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.052612:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.055337:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016143.060688:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8815949 i_blocks 96 00000020:00010000:2.0:1734016143.060695:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c3908 (ffff88012db7cc68/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.060699:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c3908 (ffff88012db7cc68/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.060705:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2c30 (ffff88012db7c1b8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.060708:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2c70 (ffff88012db7cd88/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.060711:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c3908 (ffff88012db7cc68/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.060713:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2c30 (ffff88012db7cc68/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.060718:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2c70 (ffff88012db7cc68/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.060735:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016143.060746:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8815949 00000080:00a00000:2.0:1734016143.060811:0:10366:0:(llite_mmap.c:611:ll_file_mmap()) VFS_Op: fid=[0x200000401:0xc:0x0] vma=ffff8800a9182948 start=0x7fef68fb7000 end=0x7fef68fb9000 vm_flags=0xfb 00000080:00200000:2.0:1734016143.060821:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016143.060829:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016143.060841:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.060844:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c30, osc lock ffff8800aafcf7c0, flags 0x400001002 00000008:00010000:2.0:1734016143.060849:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.060850:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c70, osc lock ffff8800aafce5f0, flags 0x400001002 00010000:00010000:2.0:1734016143.060871:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cf49 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.060877:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cf49 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.060932:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.060943:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016143.060963:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332502912:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016143.061020:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.061029:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.061035:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.065833:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00010000:00010000:2.0:1734016143.071036:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cf49 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.071063:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cf49 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.071082:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cf49 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.071088:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cf49 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.071095:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cf49 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.071116:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4621645, setting rss=4621645; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016143.071151:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cf49 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.071157:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cf49 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.071165:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cf49 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.071173:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbf840 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016143.071180:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cf49 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.071194:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016143.071203:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cf49 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.071209:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5cf49 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.071961:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332502912:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.076715:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334625024 00000100:00100000:2.0:1734016143.076736:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334625024 00000100:00100000:2.0:1734016143.076881:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14614 00000100:00100000:2.0:1734016143.076892:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341d880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334625024:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016143.076925:0:4729:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964e640/0x37cab3886ce5cf26 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4616192->4624383) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d31782da expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.077422:0:4729:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016143.077431:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341d880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334625024:12345-192.168.201.152@tcp:104: Request processed in 540us (1088us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.077440:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14614 00000080:00200000:2.0:1734016143.086991:0:4736:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016143.087076:0:4736:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1734016143.087146:0:4736:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964e640/0x37cab3886ce5cf26 lrc: 2/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4616192->4624383) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d31782da expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.087153:0:4736:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1734016143.087177:0:4736:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964e640/0x37cab3886ce5cf26 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4616192->4624383) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d31782da expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.087184:0:4736:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964e640/0x37cab3886ce5cf26 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4616192->4624383) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d31782da expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.087209:0:4736:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016143.087211:0:4736:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016143.087213:0:4736:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1734016143.087242:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332503296:192.168.201.152@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1734016143.092839:0:4729:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016143.092868:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334625152 00000100:00100000:2.0:1734016143.092878:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334625152 00000100:00100000:2.0:1734016143.092933:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14615 00000100:00100000:2.0:1734016143.092942:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012b52cc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6472:x1818247334625152:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016143.092969:0:4729:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [2547712->2551807] (req 2547712->2551807) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.092983:0:4729:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [2547712->2551807] (req 2547712->2551807) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.092996:0:4729:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.093009:0:4729:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.093020:0:4729:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.093031:0:4729:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.093067:0:4729:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a6c23840) 00000100:00100000:2.0:1734016143.093134:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012b52cc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6472:x1818247334625152:12345-192.168.201.152@tcp:105: Request processed in 192us (427us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.093142:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14615 00000100:00100000:2.0:1734016143.105044:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332503296:192.168.201.152@tcp:103:ldlm_bl.0 00000020:00010000:2.0:1734016143.168542:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724e308 (ffff88012db7cc68/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.168546:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724e308 (ffff88012db7cc68/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.168549:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f8fb0 (ffff88012db7c1b8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.168555:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.168564:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.168587:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724e308 (ffff88012db7cc68/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.168589:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f8fb0 (ffff88012db7cc68/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.168608:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016143.168658:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016143.168670:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016143.168683:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [622, 622] 00000008:00010000:2.0:1734016143.168702:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016143.168707:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8a30, osc lock ffff8800aafceab0, flags 0x0 00010000:00010000:2.0:1734016143.168727:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.168737:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2547712 2551807) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.168747:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.168753:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.168760:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.168801:0:10366:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800aa7e2d80 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016143.168818:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 1, dirtied 1 00000020:00010000:2.0:1734016143.168833:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.168835:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.168837:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8a30 (ffff88012db7cd88/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.168841:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.168846:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.168854:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.168856:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8a30 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.168864:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016143.168892:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 623 00000080:00200000:2.0:1734016143.168898:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016143.168902:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016143.168906:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [623, 623] 00000008:00010000:2.0:1734016143.168909:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016143.168912:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8a30, osc lock ffff8800aafceab0, flags 0x0 00010000:00010000:2.0:1734016143.168918:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.168931:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2551808 2555903) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.168938:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.168943:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.168949:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.168965:0:10366:0:(osc_cache.c:543:osc_extent_merge()) ### extent: ffff8800aa7e26c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.168970:0:10366:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e26c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.168976:0:10366:0:(osc_cache.c:560:osc_extent_merge()) ### extent: ffff8800aa7e2d80 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.168983:0:10366:0:(osc_cache.c:771:osc_extent_find()) ### extent: ffff8800aa7e2d80 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016143.168989:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 1, dirtied 1 00000020:00010000:2.0:1734016143.168994:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.168996:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.168998:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8a30 (ffff88012db7cd88/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.169001:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.169006:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.169012:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.169014:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8a30 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.169019:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016143.169032:0:10366:0:(file.c:5185:ll_fsync()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8), start 2547712, end 2555903, datasync 1 00000080:00200000:2.0:1734016143.169037:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016143.169041:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1734016143.169049:0:10366:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e2d80 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.169058:0:10366:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800aa7e2d80 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.169380:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2547712 2551807) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016143.169456:0:10366:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73cad80 - object 0x280000400:7 offset 2547712<>2555904 00000100:00100000:2.0:1734016143.169534:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016143.169546:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332503680:192.168.201.152@tcp:4:fsx.0 00000008:00010000:2.0:1734016143.176381:0:1971:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e2d80 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.176724:0:1971:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2d80 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.176753:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332503680:192.168.201.152@tcp:4:fsx.0 00000020:00010000:2.0:1734016143.190547:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.190552:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.190558:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f8330 (ffff88012db7c518/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.190582:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.190596:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.190617:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.190619:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f8330 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.190647:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016143.190729:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016143.190738:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016143.190764:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016143.190791:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012b714b40/0x37cab3886ce5ceee lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d317829b expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016143.190804:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012b714b40/0x37cab3886ce5ceee lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d317829b expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016143.190825:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016143.190830:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5ceee 00000080:00010000:2.0:1734016143.190835:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5ceee from it ffff8800a7343cd8 00010000:00010000:2.0:1734016143.190838:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012b714b40/0x37cab3886ce5ceee lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d317829b expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016143.190844:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012b714b40/0x37cab3886ce5ceee lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d317829b expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016143.190863:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016143.190878:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016143.190904:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.190906:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e26f0, osc lock ffff8800aafceab0, flags 0x400001002 00000008:00010000:2.0:1734016143.190913:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.190914:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2730, osc lock ffff8800aafcef70, flags 0x400001002 00010000:00010000:2.0:1734016143.190943:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5cf57 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.190950:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5cf57 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.191048:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5cf5e lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.191054:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5cf5e lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.191102:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332504320:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.191186:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016143.191193:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332504192:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.196147:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14617 00000100:00100000:2.0:1734016143.196154:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334625408:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.196163:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.196169:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.196189:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1028, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.196191:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 2612003 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 1 00000020:00010000:2.0:1734016143.196194:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 3, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.196195:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 2612003 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 1 00000100:00100000:2.0:1734016143.196257:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334625408:12345-192.168.201.152@tcp:106: Request processed in 63us (332us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.196262:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14617 00000100:00100000:2.0:1734016143.199073:0:1970:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8800b5805800: committing for last_committed 4294969894 gen 2 00010000:00010000:2.0:1734016143.199141:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5cf57 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.199159:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5cf57 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.199176:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5cf57 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.199189:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5cf57 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.199196:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5cf57 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.199258:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4621645, setting rss=4621645; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016143.199271:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5cf57 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.199279:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5cf57 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.199287:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5cf57 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.199296:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbc000 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016143.199301:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5cf57 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.199314:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016143.199321:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5cf57 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.199328:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5cf57 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.199382:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332504192:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016143.199404:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5cf5e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.199412:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5cf5e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.199420:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5cf5e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.199427:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5cf5e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.199433:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5cf5e lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.199440:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 2612003, setting rss=2612003; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016143.199459:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5cf5e lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.199464:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5cf5e lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.199470:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5cf5e lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.199481:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbf3c0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016143.199483:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5cf5e lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.199489:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016143.199491:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5cf5e lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.199497:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5cf5e lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.199514:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332504320:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.214208:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334625792 00000100:00100000:2.0:1734016143.214516:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334625792 00000100:00100000:2.0:1734016143.214535:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14621 00000100:00100000:2.0:1734016143.214539:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334625792:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.214545:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.214551:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.214561:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1028, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.214561:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4621645 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 136 00000020:00010000:2.0:1734016143.214564:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 3, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.214565:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4621645 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 136 00000100:00100000:2.0:1734016143.214605:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334625792:12345-192.168.201.152@tcp:106: Request processed in 65us (700us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.214611:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14621 00000100:00100000:2.0:1734016143.223089:0:9532:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016143.223119:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334626048 00000100:00100000:2.0:1734016143.223134:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334626048 00000100:00100000:2.0:1734016143.223206:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14622 00000100:00100000:2.0:1734016143.223256:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b35a1850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334626048:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.223277:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.223289:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.223318:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1028, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.223321:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4621645 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 136 00000020:00010000:2.0:1734016143.223324:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 3, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.223325:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4621645 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 136 00000100:00100000:2.0:1734016143.223373:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b35a1850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334626048:12345-192.168.201.152@tcp:106: Request processed in 159us (533us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.223380:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14622 00000100:00100000:2.0:1734016143.225677:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334626176 00000100:00100000:2.0:1734016143.225684:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334626176 00000100:00100000:2.0:1734016143.226390:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14623 00000100:00100000:2.0:1734016143.226400:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334626176:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.226416:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.226426:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.226467:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1028, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.226470:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 2612003 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 1 00000020:00010000:2.0:1734016143.226473:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 3, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.226474:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 2612003 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 1 00000100:00100000:2.0:1734016143.226545:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334626176:12345-192.168.201.152@tcp:106: Request processed in 146us (2444us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.226551:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14623 00000080:00200000:2.0:1734016143.229813:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8815949 i_blocks 137 00000020:00010000:2.0:1734016143.229822:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.229825:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.229830:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676330 (ffff88012db7c008/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.229834:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676370 (ffff88012db7c518/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.229837:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.229839:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676330 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.229845:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676370 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.229860:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016143.229874:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8815949 00000080:00200008:2.0:1734016143.229961:0:10366:0:(llite_lib.c:2284:ll_setattr_raw()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8) current size 8815949, valid attrs 2068, mode 8800, uid -1451891104, gid -30720, new size 3796802, atime 1734016143.000000000, mtime 1734016143.000000000, ctime 1734016143.000000000 00010000:00010000:2.0:1734016143.230008:0:10366:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb46129d8d31782cc expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016143.230034:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012b715d40/0x37cab3886ce591e6 lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3173c47 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016143.230088:0:10366:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb46129d8d31782cc expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016143.230095:0:10366:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1734016143.230097:0:10366:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5cf18 lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb46129d8d31782cc expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1734016143.230764:0:10366:0:(genops.c:2298:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8800b3cc4800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1734016143.230773:0:10366:0:(client.c:782:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8800a73c9c00 x1818247332505344/t0(0) o36->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734016143.230802:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332505344:192.168.201.152@tcp:36:fsx.0 00000100:00100000:2.0:1734016143.230886:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e20c0 going to sleep for 11 seconds 00000100:00100000:2.0:1734016143.234697:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016143.239979:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332505344:192.168.201.152@tcp:36:fsx.0 00000080:00200000:2.0:1734016143.240036:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:2.0:1734016143.240046:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:2.0:1734016143.240068:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016143.240089:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1734016143.240106:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016143.240111:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e20f0, osc lock ffff8800aafce4c0, flags 0x0 00000008:00010000:2.0:1734016143.240114:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016143.240116:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2130, osc lock ffff8800aafcf1d0, flags 0x0 00010000:00010000:2.0:1734016143.240141:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.240160:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.240172:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.240180:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.240187:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.240200:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.240208:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.240215:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.240240:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.240246:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.240301:0:10366:0:(osc_cache.c:2696:osc_cache_truncate_start()) ### extent: ffff8800a708c180 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016143.240317:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016143.240596:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:2.0:1734016143.240603:0:10366:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a708c180 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016143.243381:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 3796802 i_blocks 112 00000020:00010000:2.0:1734016143.243388:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.243392:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.243395:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e20f0 (ffff88012db7c518/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.243408:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.243424:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.243444:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2130 (ffff88012db7c008/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.243449:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.243456:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.243468:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.243470:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e20f0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.243475:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2130 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.243500:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016143.243521:0:10366:0:(llite_lib.c:2534:ll_setattr_raw()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8) current size 3796802, valid attrs 2168, mode 8800, uid -1451891104, gid -30720, new size 3796802, atime 1734016143.000000000, mtime 1734016143.229000000, ctime 1734016143.229000000, rc 0 00000080:00200008:2.0:1734016143.243556:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016143.243563:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016143.243598:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016143.243618:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b62b8800 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:2.0:1734016143.243679:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cf8f lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016143.243686:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cf8f lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016143.243698:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5cf8f lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00000100:00100000:2.0:1734016143.243727:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8700 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332505856:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.243803:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e20c0 going to sleep for 11 seconds 00000080:00200000:2.0:1734016143.283973:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 5608527 i_blocks 112 00000020:00010000:2.0:1734016143.283986:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ec68 (ffff88012db7cd88/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.283990:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ec68 (ffff88012db7cd88/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.283998:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86763f0 (ffff88012db7cc68/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.284014:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.284032:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.284054:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676430 (ffff88012db7c488/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.284060:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.284067:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.284074:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ec68 (ffff88012db7cd88/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.284076:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86763f0 (ffff88012db7cd88/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.284084:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676430 (ffff88012db7cd88/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.284140:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016143.284166:0:10366:0:(llite_lib.c:2534:ll_setattr_raw()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8) current size 5608527, valid attrs 2168, mode 8800, uid -1451891104, gid -30720, new size 5608527, atime 1734016143.000000000, mtime 1734016143.250000000, ctime 1734016143.250000000, rc 0 00000080:00a00000:2.0:1734016143.284205:0:10366:0:(llite_mmap.c:611:ll_file_mmap()) VFS_Op: fid=[0x200000401:0xc:0x0] vma=ffff8800a9182e58 start=0x7fef68fb6000 end=0x7fef68fb9000 vm_flags=0xfb 00000080:00200000:2.0:1734016143.284245:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016143.284254:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016143.284270:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.284272:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2270, osc lock ffff8800aafcf1d0, flags 0x400001002 00000008:00010000:2.0:1734016143.284279:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.284280:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e22b0, osc lock ffff8800aafce4c0, flags 0x400001002 00010000:00010000:2.0:1734016143.284297:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.284313:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.284323:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.284329:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.284337:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.284348:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.284354:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.284359:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016143.284368:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 5608527 i_blocks 112 00000020:00010000:2.0:1734016143.284371:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.284373:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.284375:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2270 (ffff88012db7c008/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.284377:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e22b0 (ffff88012db7c518/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.284379:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.284381:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2270 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.284385:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e22b0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.284395:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016143.284459:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 1367 00000080:00200000:2.0:1734016143.284467:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016143.284477:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016143.284490:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1367, 1367] 00000008:00010000:2.0:1734016143.284497:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016143.284500:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8db0, osc lock ffff8800aafce4c0, flags 0x0 00010000:00010000:2.0:1734016143.284507:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.284516:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1404928 1409023) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.284522:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.284528:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.284533:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.284580:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.284591:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.284598:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 7/1,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.284604:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 7/1,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.284609:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 6/1,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.286388:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 6/1,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.286404:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.342686:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.342691:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.342695:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8db0 (ffff88012db7c518/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.342707:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.342720:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.342744:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.342747:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8db0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.342776:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016143.342796:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016143.342814:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016143.342828:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1367, 1367] 00000008:00010000:2.0:1734016143.342842:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016143.342848:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8db0, osc lock ffff8800aafce4c0, flags 0x0 00010000:00010000:2.0:1734016143.342869:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.342879:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1404928 1409023) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.342888:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.342894:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.342898:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.342943:0:10366:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800aa7e2240 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016143.342964:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 1, dirtied 1 00000020:00010000:2.0:1734016143.342976:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.342977:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.342980:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8db0 (ffff88012db7c518/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.342983:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.342988:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.342995:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.342997:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8db0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.343004:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016143.343040:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 1368 00000080:00200000:2.0:1734016143.343047:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016143.343052:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016143.343057:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1368, 1368] 00000008:00010000:2.0:1734016143.343061:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016143.343063:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8db0, osc lock ffff8800aafce4c0, flags 0x0 00010000:00010000:2.0:1734016143.343068:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.343074:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1409024 1413119) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.343081:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.343090:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.343096:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.343118:0:10366:0:(osc_cache.c:543:osc_extent_merge()) ### extent: ffff8800aa7e20c0 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.343125:0:10366:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e20c0 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.343131:0:10366:0:(osc_cache.c:560:osc_extent_merge()) ### extent: ffff8800aa7e2240 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.343139:0:10366:0:(osc_cache.c:771:osc_extent_find()) ### extent: ffff8800aa7e2240 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016143.343148:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 1, dirtied 1 00000020:00010000:2.0:1734016143.343153:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.343155:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.343157:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8db0 (ffff88012db7c518/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.343160:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.343165:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.343172:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.343174:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8db0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.343179:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016143.343190:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 1369 00000080:00200000:2.0:1734016143.343193:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016143.343197:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016143.343200:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1369, 1369] 00000008:00010000:2.0:1734016143.343204:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016143.343206:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8db0, osc lock ffff8800aafce4c0, flags 0x0 00010000:00010000:2.0:1734016143.343211:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.343241:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1413120 1417215) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.343248:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.343254:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.343260:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000001:00010000:2.0:1734016143.343271:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016143.343280:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.343281:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e20f0, osc lock ffff8800aafcf1d0, flags 0x400001002 00000008:00010000:2.0:1734016143.343284:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.343285:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2130, osc lock ffff8800aafce850, flags 0x400001002 00010000:00010000:2.0:1734016143.343295:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.343303:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.343310:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.343318:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.343331:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.343339:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.343346:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 8/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.343352:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 7/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016143.343363:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 5608527 i_blocks 112 00000020:00010000:2.0:1734016143.343367:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.343370:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.343372:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e20f0 (ffff88012db7c008/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.343374:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2130 (ffff88012db7c518/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.343376:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.343378:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e20f0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.343381:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2130 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000008:00010000:2.0:1734016143.343391:0:10366:0:(osc_cache.c:543:osc_extent_merge()) ### extent: ffff8800aa7e20c0 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.343397:0:10366:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e20c0 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.343403:0:10366:0:(osc_cache.c:560:osc_extent_merge()) ### extent: ffff8800aa7e2240 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.343410:0:10366:0:(osc_cache.c:771:osc_extent_find()) ### extent: ffff8800aa7e2240 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016143.343418:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 1, dirtied 1 00000020:00010000:2.0:1734016143.343425:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.343427:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.343429:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8db0 (ffff88012db7c518/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.343432:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.343439:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.343452:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.343454:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8db0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.343465:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016143.343480:0:10366:0:(file.c:5185:ll_fsync()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8), start 5599232, end 5611519, datasync 1 00000080:00200000:2.0:1734016143.343486:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016143.343491:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1734016143.343501:0:10366:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e2240 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.343514:0:10366:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800aa7e2240 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.343665:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1404928 1409023) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016143.343769:0:10366:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73cb100 - object 0x240000400:7 offset 1404928<>1414223 00000100:00100000:2.0:1734016143.343820:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016143.343840:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332506752:192.168.201.152@tcp:4:fsx.0 00000080:00200000:2.0:1734016143.350453:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1734016143.350510:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8700 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332506880:192.168.201.152@tcp:44:fsx.0 00000100:00100000:2.0:1734016143.350578:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2240 going to sleep for 11 seconds 00000100:00100000:2.0:1734016143.394889:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016143.395700:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016143.395747:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334626432 00000100:00100000:2.0:1734016143.395765:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334626432 00000100:00100000:2.0:1734016143.395832:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334626560 00000100:00100000:2.0:1734016143.395835:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334626560 00000100:00100000:2.0:1734016143.395855:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14625 00000100:00100000:2.0:1734016143.395868:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73cad80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334626432:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.395892:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.395907:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.395955:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1274, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.395959:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 1414223 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 24 00000020:00010000:2.0:1734016143.395962:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.395963:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 1414223 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 24 00000100:00100000:2.0:1734016143.396045:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73cad80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334626432:12345-192.168.201.152@tcp:106: Request processed in 181us (1179us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.396052:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14625 00000100:00100000:2.0:1734016143.396080:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14626 00000100:00100000:2.0:1734016143.396084:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73c9f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334626560:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.396094:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.396103:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.396117:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1274, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.396119:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 112 00000020:00010000:2.0:1734016143.396123:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 3, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.396124:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 112 00000100:00100000:2.0:1734016143.396145:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73c9f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334626560:12345-192.168.201.152@tcp:106: Request processed in 58us (445us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.396151:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14626 00000080:00200000:2.0:1734016143.404383:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 5608527 i_blocks 136 00000020:00010000:2.0:1734016143.404391:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1da8 (ffff88012db7cab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.404394:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1da8 (ffff88012db7cab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.404402:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676e70 (ffff88012db7c368/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.404407:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676eb0 (ffff88012db7c908/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.404410:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1da8 (ffff88012db7cab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.404413:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676e70 (ffff88012db7cab8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.404419:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676eb0 (ffff88012db7cab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.404514:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016143.404562:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016143.404578:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016143.404589:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016143.404611:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.404616:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e20f0, osc lock ffff8800aafce4c0, flags 0x400001002 00000008:00010000:2.0:1734016143.404625:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.404626:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2130, osc lock ffff8800aafce850, flags 0x400001002 00010000:00010000:2.0:1734016143.404656:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cfab lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.404664:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cfab lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.404751:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5cfb2 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.404758:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5cfb2 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.404806:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332507648:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.404890:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016143.404894:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332507520:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.411976:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016143.412052:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016143.412122:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334626688 00000100:00100000:2.0:1734016143.412137:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334626688 00000100:00100000:2.0:1734016143.412189:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334626816 00000100:00100000:2.0:1734016143.412192:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334626816 00000100:00100000:2.0:1734016143.412212:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14627 00000100:00100000:2.0:1734016143.415075:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73cbb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334626688:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.415113:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.415133:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.415184:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1274, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.415186:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 1414223 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 24 00000020:00010000:2.0:1734016143.415189:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.415190:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 1414223 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 24 00000100:00100000:2.0:1734016143.415316:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73cbb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334626688:12345-192.168.201.152@tcp:106: Request processed in 275us (3358us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.415700:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14627 00000080:00200000:2.0:1734016143.425579:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 5608527 i_blocks 136 00000020:00010000:2.0:1734016143.425587:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.425592:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.425601:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e20f0 (ffff88012db7c008/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.425608:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2130 (ffff88012db7c518/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.425611:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.425612:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e20f0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.425621:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2130 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.425645:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016143.425663:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 5608527 00000080:00200008:2.0:1734016143.425776:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 4150579, type: SEEK_SET 00000080:00200008:2.0:1734016143.425789:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 4150579, type: SEEK_SET, rc = 4150579 00000080:00200008:2.0:1734016143.425808:0:10366:0:(file.c:2290:do_file_read_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 4150579, count: 5309 00000080:00200000:2.0:1734016143.425817:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016143.425837:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016143.425903:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 1013 00000080:00200000:2.0:1734016143.425915:0:10366:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: read ppos: 4150579, bytes: 5309 00000080:00200000:2.0:1734016143.425924:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:2.0:1734016143.425934:0:10366:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b62c6a28[0x4, 1274, [0x200000401:0xc:0x0] hash] 00000020:00200000:2.0:1734016143.425937:0:10366:0:(cl_io.c:195:cl_io_rw_init()) io range: 1 [4150579, 4155888) 0 0 00000080:00200000:2.0:1734016143.425939:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016143.425963:0:10366:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 0 chunk: [4150579, 4155888) 4155888, 5309 00020000:00200000:2.0:1734016143.425969:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016143.426005:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [1013, 1014] 00000008:00010000:2.0:1734016143.426030:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016143.426035:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8db0, osc lock ffff8800aafce850, flags 0x0 00010000:00010000:2.0:1734016143.426071:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.426086:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4149248 4157439) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.426099:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 7/1,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.426105:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 7/1,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.426110:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 6/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016143.426116:0:10366:0:(vvp_io.c:840:vvp_io_read_start()) f16b.sanityn: read [4150579, 4155888) 00000080:00200000:2.0:1734016143.426138:0:10366:0:(vvp_io.c:899:vvp_io_read_start()) count:5309 iocb pos:4150579 00010000:00010000:2.0:1734016143.426201:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/1,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.426209:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/1,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.426252:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 8/2,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.426260:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 8/2,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.426270:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 7/2,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.427624:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 7/2,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.427642:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 6/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.448287:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.451532:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.451583:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/1,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.451591:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 5/1,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.451600:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.451846:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.451857:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.451944:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.451948:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.451956:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8db0 (ffff88012db7c518/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.451966:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.451973:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.451983:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ef38 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.451985:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8db0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.452011:0:10366:0:(vvp_io.c:533:vvp_io_advance()) advancing 5309 bytes 00000080:00200000:2.0:1734016143.452044:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016143.452048:0:10366:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 1 io complete with rc: 0, result: 5309, restart: 0 00000080:00200000:2.0:1734016143.452056:0:10366:0:(file.c:2044:ll_file_io_generic()) iot: 1, result: 5309 00000080:00200008:2.0:1734016143.452102:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016143.452110:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016143.452144:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016143.452173:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cf96 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178366 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016143.452185:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cf96 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178366 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016143.452205:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016143.452209:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5cf96 00000080:00010000:2.0:1734016143.452240:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5cf96 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016143.452244:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cf96 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178366 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016143.452252:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5cf96 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178366 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016143.452290:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016143.452305:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016143.452332:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.452335:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2db0, osc lock ffff8800aafce850, flags 0x400001002 00000008:00010000:2.0:1734016143.452341:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.452343:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2df0, osc lock ffff8800aafce4c0, flags 0x400001002 00010000:00010000:2.0:1734016143.452377:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf180/0x37cab3886ce5cfb9 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.452387:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf180/0x37cab3886ce5cfb9 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.452544:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016143.452589:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332508032:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016143.452999:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5cfc0 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.453007:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5cfc0 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.453053:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332508160:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.466184:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016143.466321:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016143.474326:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 5608527 i_blocks 136 00000020:00010000:2.0:1734016143.474332:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.474335:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.474342:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2db0 (ffff88012db7c008/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.474347:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2df0 (ffff88012db7c518/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.474350:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.474352:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2db0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.474357:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2df0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.474372:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016143.474399:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016143.474406:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016143.474417:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016143.474434:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.474437:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2db0, osc lock ffff8800aafce4c0, flags 0x400001002 00000008:00010000:2.0:1734016143.474442:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.474444:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2df0, osc lock ffff8800aafce850, flags 0x400001002 00010000:00010000:2.0:1734016143.474474:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cfc7 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.474483:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5cfc7 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.474574:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cfce lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.474581:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cfce lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.477109:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016143.477162:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00010000:00010000:2.0:1734016143.483709:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cfce lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.483731:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cfce lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.483741:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cfce lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.483747:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cfce lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.483753:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cfce lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.483768:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 1414223, setting rss=1414223; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016143.483842:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cfce lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.483848:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cfce lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.483859:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cfce lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.483865:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbdf80 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016143.483869:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cfce lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.483879:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016143.483884:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cfce lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.483889:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5cfce lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.483925:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332508416:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.488697:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334627456 00000100:00100000:2.0:1734016143.488714:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334627456 00000100:00100000:2.0:1734016143.488789:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14632 00000100:00100000:2.0:1734016143.488798:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334627456:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.488817:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.488828:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.488869:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1371, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.488873:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 112 00000020:00010000:2.0:1734016143.488876:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 3, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.488877:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 112 00000100:00100000:2.0:1734016143.488944:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334627456:12345-192.168.201.152@tcp:106: Request processed in 147us (429us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.488952:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14632 00000100:00100000:2.0:1734016143.489679:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334627584 00000100:00100000:2.0:1734016143.489683:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334627584 00000100:00100000:2.0:1734016143.489711:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14633 00000100:00100000:2.0:1734016143.489716:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334627584:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.489724:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.489732:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.489743:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1371, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.489745:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 1414223 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 24 00000020:00010000:2.0:1734016143.489748:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.489749:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 1414223 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 24 00000100:00100000:2.0:1734016143.489770:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334627584:12345-192.168.201.152@tcp:106: Request processed in 53us (187us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.489776:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14633 00000100:00100000:2.0:1734016143.499340:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334627712 00000100:00100000:2.0:1734016143.499355:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334627712 00000100:00100000:2.0:1734016143.499409:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14634 00000100:00100000:2.0:1734016143.499416:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334627712:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.499433:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.499447:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.499484:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1371, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.499487:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 112 00000020:00010000:2.0:1734016143.499490:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 3, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.499492:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 112 00000100:00100000:2.0:1734016143.499549:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334627712:12345-192.168.201.152@tcp:106: Request processed in 132us (607us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.499557:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14634 00000100:00100000:2.0:1734016143.500309:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334627840 00000100:00100000:2.0:1734016143.500316:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334627840 00000100:00100000:2.0:1734016143.500484:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14635 00000100:00100000:2.0:1734016143.500491:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334627840:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.500501:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.500511:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.500527:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1371, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.500531:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 1414223 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 24 00000020:00010000:2.0:1734016143.500534:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.500536:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 1414223 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 24 00000100:00100000:2.0:1734016143.500579:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334627840:12345-192.168.201.152@tcp:106: Request processed in 90us (690us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.500585:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14635 00000100:00100000:2.0:1734016143.512725:0:4729:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016143.512749:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334627968 00000100:00100000:2.0:1734016143.512762:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334627968 00000100:00100000:2.0:1734016143.512814:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14636 00000100:00100000:2.0:1734016143.512823:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012b52e050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334627968:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016143.512843:0:4729:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c22d00/0x37cab3886ce5cee7 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d317828d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.512931:0:4729:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016143.512937:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012b52e050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334627968:12345-192.168.201.152@tcp:104: Request processed in 115us (345us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.512943:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14636 00000100:00100000:2.0:1734016143.530412:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334628096 00000100:00100000:2.0:1734016143.530429:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334628096 00000100:00100000:2.0:1734016143.533475:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334628224 00000100:00100000:2.0:1734016143.533484:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334628224 00000100:00100000:2.0:1734016143.533521:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14638 00000100:00100000:2.0:1734016143.533529:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334628224:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.533545:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.533556:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.533600:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.533602:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 112 00000020:00010000:2.0:1734016143.533606:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 3, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.533607:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 112 00000100:00100000:2.0:1734016143.533668:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334628224:12345-192.168.201.152@tcp:106: Request processed in 139us (273us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.533677:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14638 00000080:00200000:2.0:1734016143.535354:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7730817 i_blocks 136 00000020:00010000:2.0:1734016143.535361:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.535364:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.535368:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676330 (ffff88012db7c008/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.535372:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676370 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.535375:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.535377:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676330 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.535382:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676370 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.535396:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016143.535432:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016143.535440:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016143.535452:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016143.535469:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.535471:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e29f0, osc lock ffff8800aafcf1d0, flags 0x400001002 00000008:00010000:2.0:1734016143.535478:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.535479:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2a30, osc lock ffff8800aafcf430, flags 0x400001002 00010000:00010000:2.0:1734016143.535505:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5cfff lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.535518:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5cfff lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.535613:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.535629:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016143.535644:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332509440:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016143.535724:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.535735:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.535741:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.537912:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334628352 00000100:00100000:2.0:1734016143.537930:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334628352 00000100:00100000:2.0:1734016143.538003:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14639 00000100:00100000:2.0:1734016143.538015:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334628352:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.538037:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.538052:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.538112:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.538115:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 112 00000020:00010000:2.0:1734016143.538118:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 3, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.538119:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 112 00000100:00100000:2.0:1734016143.538204:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334628352:12345-192.168.201.152@tcp:106: Request processed in 190us (380us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.538213:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14639 00010000:00010000:2.0:1734016143.539997:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5cfff lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.540016:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5cfff lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.540031:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5cfff lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.540039:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5cfff lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.540046:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5cfff lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.540067:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016143.540176:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5cfff lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.540191:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5cfff lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.540199:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5cfff lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.540206:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbdd40 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016143.540212:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5cfff lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.540247:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016143.540254:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5cfff lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.540260:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5cfff lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.540302:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332509440:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.547185:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334628480 00000100:00100000:2.0:1734016143.547197:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334628480 00000100:00100000:2.0:1734016143.547644:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14640 00000100:00100000:2.0:1734016143.547658:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334628480:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.547680:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.547700:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.547747:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 2, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.547752:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 3536513 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 24 00000020:00010000:2.0:1734016143.547756:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 3, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.547757:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 3536513 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 24 00000100:00100000:2.0:1734016143.548541:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334628480:12345-192.168.201.152@tcp:106: Request processed in 883us (1492us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.548550:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14640 00000080:00200000:2.0:1734016143.553169:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7730817 i_blocks 136 00000020:00010000:2.0:1734016143.553179:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.553183:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.553189:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86764b0 (ffff88012db7c008/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.553195:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86764f0 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.553198:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c3b50 (ffff88012db7c1b8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.553200:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86764b0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.553207:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86764f0 (ffff88012db7c1b8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.553235:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016143.553257:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016143.553262:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016143.553271:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016143.553286:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.553288:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2cf0, osc lock ffff8800aafce130, flags 0x400001002 00000008:00010000:2.0:1734016143.553293:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.553295:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2d30, osc lock ffff8800aafce260, flags 0x400001002 00010000:00010000:2.0:1734016143.553312:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.553323:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.553331:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.553337:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.553359:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d014 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.553364:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d014 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.553416:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332509824:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.556629:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334628608 00000100:00100000:2.0:1734016143.556639:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334628608 00000100:00100000:2.0:1734016143.556693:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14641 00000100:00100000:2.0:1734016143.556701:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334628608:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.556718:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.556728:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.556759:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 2, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.556762:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 3536513 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 24 00000020:00010000:2.0:1734016143.556764:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 3, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.556765:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 3536513 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 24 00000100:00100000:2.0:1734016143.556828:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334628608:12345-192.168.201.152@tcp:106: Request processed in 127us (288us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.556834:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14641 00010000:00010000:2.0:1734016143.559018:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d014 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.559047:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d014 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.559061:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d014 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.559071:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d014 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.559079:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d014 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.559101:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3536513, setting rss=3536513; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016143.559159:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d014 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.559173:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d014 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.559183:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d014 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.559191:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbdd40 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016143.559198:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d014 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.559209:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016143.559215:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d014 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.559255:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d014 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.559363:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332509824:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.563512:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334628736 00000100:00100000:2.0:1734016143.563526:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334628736 00000100:00100000:2.0:1734016143.563587:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14642 00000100:00100000:2.0:1734016143.563596:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334628736:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.563612:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.563630:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.563671:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.563674:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 112 00000020:00010000:2.0:1734016143.563678:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 3, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.563679:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 112 00000100:00100000:2.0:1734016143.563738:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334628736:12345-192.168.201.152@tcp:106: Request processed in 144us (405us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.563745:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14642 00000100:00100000:2.0:1734016143.571548:0:9532:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016143.571586:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334628864 00000100:00100000:2.0:1734016143.571605:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334628864 00000100:00100000:2.0:1734016143.571683:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14643 00000100:00100000:2.0:1734016143.571691:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7039c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334628864:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.571712:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.571723:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.571761:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.571764:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 112 00000020:00010000:2.0:1734016143.571768:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 3, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.571768:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 112 00000100:00100000:2.0:1734016143.571839:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7039c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334628864:12345-192.168.201.152@tcp:106: Request processed in 147us (723us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.571846:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14643 00000100:00100000:2.0:1734016143.580632:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334628992 00000100:00100000:2.0:1734016143.580647:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334628992 00000100:00100000:2.0:1734016143.580918:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14644 00000100:00100000:2.0:1734016143.580928:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334628992:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.580951:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.580965:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.581005:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 2, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.581008:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 3536513 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 24 00000020:00010000:2.0:1734016143.581010:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 3, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.581011:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 3536513 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 24 00000100:00100000:2.0:1734016143.581330:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334628992:12345-192.168.201.152@tcp:106: Request processed in 401us (1411us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.581340:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14644 00000020:00010000:2.0:1734016143.589535:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724e7b8 (ffff88012db7c518/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.589540:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724e7b8 (ffff88012db7c518/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.589546:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b352ef30 (ffff88012db7cb48/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.589569:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.589581:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.589596:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724e7b8 (ffff88012db7c518/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.589598:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b352ef30 (ffff88012db7c518/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.589617:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016143.589688:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016143.589698:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016143.589723:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016143.589746:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff8800a6c23600/0x37cab3886ce5d006 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31783dd expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016143.589761:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff8800a6c23600/0x37cab3886ce5d006 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31783dd expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016143.589781:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016143.589784:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d006 00000080:00010000:2.0:1734016143.589792:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d006 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016143.589797:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff8800a6c23600/0x37cab3886ce5d006 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31783dd expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016143.589803:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff8800a6c23600/0x37cab3886ce5d006 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31783dd expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016143.589831:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016143.589847:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016143.589867:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.589870:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e23f0, osc lock ffff8800aafce000, flags 0x400001002 00000008:00010000:2.0:1734016143.589877:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.589878:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2430, osc lock ffff8800aafcebe0, flags 0x400001002 00010000:00010000:2.0:1734016143.589896:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.589903:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.589914:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.589922:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.589953:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d030 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.589958:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d030 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.590497:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016143.590519:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332510464:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.593575:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334629120 00000100:00100000:2.0:1734016143.593588:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334629120 00000100:00100000:2.0:1734016143.593639:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14645 00000100:00100000:2.0:1734016143.593646:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334629120:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.593660:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.593672:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.593701:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 2, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.593704:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 3536513 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 24 00000020:00010000:2.0:1734016143.593707:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 3, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.593708:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 3536513 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 24 00000100:00100000:2.0:1734016143.593752:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334629120:12345-192.168.201.152@tcp:106: Request processed in 106us (297us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.593759:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14645 00010000:00010000:2.0:1734016143.595630:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d030 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.595653:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d030 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.595671:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d030 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.595680:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d030 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.595688:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d030 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.595713:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3536513, setting rss=3536513; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016143.595919:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d030 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.595932:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d030 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.595940:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d030 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.595948:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbcfc0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016143.595955:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d030 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.595969:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016143.595981:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d030 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.595987:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d030 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.596399:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332510464:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.601304:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334629248 00000100:00100000:2.0:1734016143.601321:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334629248 00000100:00100000:2.0:1734016143.601412:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14646 00000100:00100000:2.0:1734016143.601421:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ca80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334629248:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.601442:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.601461:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.601509:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 2, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.601511:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 3536513 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 24 00000020:00010000:2.0:1734016143.601514:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 3, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.601515:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 3536513 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 24 00000100:00100000:2.0:1734016143.601903:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ca80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334629248:12345-192.168.201.152@tcp:106: Request processed in 480us (760us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.601912:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14646 00000080:00200000:2.0:1734016143.611378:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7730817 i_blocks 152 00000020:00010000:2.0:1734016143.611389:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c0248 (ffff88012db7c518/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.611393:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c0248 (ffff88012db7c518/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.611399:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676b70 (ffff88012db7c128/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.611406:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676bb0 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.611409:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c0248 (ffff88012db7c518/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.611411:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676b70 (ffff88012db7c518/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.611419:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676bb0 (ffff88012db7c518/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.611433:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016143.611451:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7730817 00000080:00a00000:2.0:1734016143.611579:0:10366:0:(llite_mmap.c:611:ll_file_mmap()) VFS_Op: fid=[0x200000401:0xc:0x0] vma=ffff8800a9182000 start=0x7fef68fb7000 end=0x7fef68fb9000 vm_flags=0xfb 00000080:00200000:2.0:1734016143.611594:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016143.611606:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016143.611629:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.611633:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2630, osc lock ffff8800aafce390, flags 0x400001002 00000008:00010000:2.0:1734016143.611644:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.611645:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2670, osc lock ffff8800aafcebe0, flags 0x400001002 00010000:00010000:2.0:1734016143.611681:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d03e lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.611691:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d03e lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.611781:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.611806:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332510720:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016143.611893:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.611904:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.611910:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.615144:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334629376 00000100:00100000:2.0:1734016143.615159:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334629376 00000100:00100000:2.0:1734016143.615848:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14647 00000100:00100000:2.0:1734016143.615859:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334629376:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.615885:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.615907:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.615950:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.615953:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 128 00000020:00010000:2.0:1734016143.615957:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 3, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.615958:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 128 00000100:00100000:2.0:1734016143.616051:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334629376:12345-192.168.201.152@tcp:106: Request processed in 193us (1250us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.616062:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14647 00010000:00010000:2.0:1734016143.620082:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d03e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.620115:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d03e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.620132:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d03e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.620138:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d03e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.620147:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d03e lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.620175:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016143.620199:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d03e lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.620206:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d03e lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.620247:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d03e lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.620257:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbf3c0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016143.620263:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d03e lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.620279:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016143.620287:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d03e lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.620294:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d03e lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.620329:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332510720:192.168.201.152@tcp:101:fsx.0 00000020:00010000:2.0:1734016143.676268:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724e7b8 (ffff88012db7c518/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.676272:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724e7b8 (ffff88012db7c518/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.676276:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f87b0 (ffff88012db7c128/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.676284:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.676300:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.676317:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724e7b8 (ffff88012db7c518/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.676318:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f87b0 (ffff88012db7c518/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.676337:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016143.676350:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016143.676362:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016143.676375:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1843, 1843] 00000008:00010000:2.0:1734016143.676389:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016143.676393:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8db0, osc lock ffff8800aafce000, flags 0x0 00010000:00010000:2.0:1734016143.676408:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.676419:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3354624 3358719) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.676426:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 8/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.676431:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.676437:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.676475:0:10366:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800aa7e2cc0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016143.676486:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af698248, count 1, dirtied 1 00000020:00010000:2.0:1734016143.676495:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724e7b8 (ffff88012db7c518/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.676498:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724e7b8 (ffff88012db7c518/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.676500:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8db0 (ffff88012db7c128/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.676502:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.676508:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.676513:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724e7b8 (ffff88012db7c518/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.676515:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8db0 (ffff88012db7c518/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.676524:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016143.676553:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 1844 00000080:00200000:2.0:1734016143.676570:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016143.676573:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016143.676576:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1844, 1844] 00000008:00010000:2.0:1734016143.676580:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016143.676581:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8db0, osc lock ffff8800aafce000, flags 0x0 00010000:00010000:2.0:1734016143.676586:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.676593:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3358720 3362815) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.676604:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 8/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.676610:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 8/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.676615:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 7/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.676630:0:10366:0:(osc_cache.c:543:osc_extent_merge()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.676636:0:10366:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.676641:0:10366:0:(osc_cache.c:560:osc_extent_merge()) ### extent: ffff8800aa7e2cc0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.676647:0:10366:0:(osc_cache.c:771:osc_extent_find()) ### extent: ffff8800aa7e2cc0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016143.676656:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af698248, count 1, dirtied 1 00000020:00010000:2.0:1734016143.676660:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724e7b8 (ffff88012db7c518/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.676661:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724e7b8 (ffff88012db7c518/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.676663:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8db0 (ffff88012db7c128/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.676666:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.676674:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.676681:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724e7b8 (ffff88012db7c518/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.676682:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8db0 (ffff88012db7c518/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.676686:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016143.676700:0:10366:0:(file.c:5185:ll_fsync()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8), start 7548928, end 7557119, datasync 1 00000080:00200000:2.0:1734016143.676706:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016143.676710:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1734016143.676719:0:10366:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e2cc0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.676726:0:10366:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a8676d80 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.676735:0:10366:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800aa7e2cc0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.678990:0:10366:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800a8676d80 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.679062:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3354624 3358719) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016143.679182:0:10366:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73c8a80 - object 0x240000400:7 offset 3354624<>3536513 00000008:00010000:2.0:1734016143.695851:0:1970:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e2cc0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.695904:0:1970:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2cc0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.695913:0:1970:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800a8676d80 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.695925:0:1970:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a8676d80 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.695953:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332511104:192.168.201.152@tcp:4:fsx.0 00000080:00200000:2.0:1734016143.705201:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7730817 i_blocks 176 00000080:00200000:2.0:1734016143.705288:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1734016143.716161:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334629504 00000100:00100000:2.0:1734016143.716176:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334629504 00000100:00100000:2.0:1734016143.716597:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14648 00000100:00100000:2.0:1734016143.716607:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334629504:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.716627:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.716639:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.716673:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 865, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.716676:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 3536513 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 48 00000020:00010000:2.0:1734016143.716679:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.716681:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 3536513 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 48 00000100:00100000:2.0:1734016143.716738:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334629504:12345-192.168.201.152@tcp:106: Request processed in 133us (705us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.716745:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14648 00010000:00010000:2.0:1734016143.721086:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964e640/0x37cab3886ce5d045 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.721105:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964e640/0x37cab3886ce5d045 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.721115:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964e640/0x37cab3886ce5d045 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.721120:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964e640/0x37cab3886ce5d045 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.721125:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964e640/0x37cab3886ce5d045 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.721136:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3536513, setting rss=3536513; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016143.722269:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964e640/0x37cab3886ce5d045 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.722279:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964e640/0x37cab3886ce5d045 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.722288:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964e640/0x37cab3886ce5d045 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.722297:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a964e640 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016143.722302:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964e640/0x37cab3886ce5d045 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.722314:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016143.722320:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964e640/0x37cab3886ce5d045 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.722325:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964e640/0x37cab3886ce5d045 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.724519:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332511360:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.735100:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334629632 00000100:00100000:2.0:1734016143.735112:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334629632 00000100:00100000:2.0:1734016143.735180:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14649 00000100:00100000:2.0:1734016143.735189:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334629632:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.735211:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.735244:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.735286:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 865, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.735288:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 3536513 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 48 00000020:00010000:2.0:1734016143.735292:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.735294:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 3536513 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 48 00000100:00100000:2.0:1734016143.735366:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334629632:12345-192.168.201.152@tcp:106: Request processed in 176us (977us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.735375:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14649 00010000:00010000:2.0:1734016143.743352:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5d04c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.743389:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5d04c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.743408:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5d04c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.743417:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5d04c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.743425:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5d04c lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.743448:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3536513, setting rss=3536513; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016143.743488:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5d04c lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.743494:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5d04c lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.743502:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5d04c lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.743512:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a964db00 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016143.743520:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5d04c lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.743534:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016143.743540:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5d04c lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.743545:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964db00/0x37cab3886ce5d04c lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.743608:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332511488:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.752150:0:9532:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016143.752179:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334629760 00000100:00100000:2.0:1734016143.752187:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334629760 00000100:00100000:2.0:1734016143.752248:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14650 00000100:00100000:2.0:1734016143.752256:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a703b450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334629760:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.752273:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.752284:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.752310:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.752312:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 128 00000020:00010000:2.0:1734016143.752315:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 3, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.752316:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 128 00000100:00100000:2.0:1734016143.752377:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a703b450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334629760:12345-192.168.201.152@tcp:106: Request processed in 119us (1978us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.752383:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14650 00000100:00100000:2.0:1734016143.769997:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016143.788439:0:4736:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016143.792724:0:4736:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1734016143.792810:0:4736:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 2/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.792823:0:4736:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1734016143.792887:0:4736:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.792897:0:4736:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5cf50 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2547712->2551807) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d3178304 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.792924:0:4736:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016143.792926:0:4736:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016143.792931:0:4736:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1734016143.792948:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016143.792978:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332512000:192.168.201.152@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1734016143.809323:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016143.824005:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332512000:192.168.201.152@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1734016143.844068:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016143.844146:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016143.844200:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334630144 00000100:00100000:2.0:1734016143.844210:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334630144 00000100:00100000:2.0:1734016143.844278:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334630272 00000100:00100000:2.0:1734016143.844281:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334630272 00000100:00100000:2.0:1734016143.844291:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14653 00000100:00100000:2.0:1734016143.844300:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73cb100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7694:x1818247334630144:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.844319:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.844330:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.844361:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 865, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.844363:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 168 00000020:00010000:2.0:1734016143.844366:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.844367:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 168 00000100:00100000:2.0:1734016143.844421:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73cb100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7694:x1818247334630144:12345-192.168.201.152@tcp:106: Request processed in 122us (370us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.844427:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14653 00000100:00100000:2.0:1734016143.844446:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14654 00000100:00100000:2.0:1734016143.844450:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73c8a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334630272:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.844457:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.844464:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.844477:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 865, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.844478:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 3536513 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 48 00000020:00010000:2.0:1734016143.844482:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.844483:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 3536513 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 48 00000100:00100000:2.0:1734016143.844497:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73c8a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334630272:12345-192.168.201.152@tcp:106: Request processed in 46us (351us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.844501:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14654 00000100:00100000:2.0:1734016143.866943:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016143.878069:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016143.886147:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7730817 i_blocks 216 00000020:00010000:2.0:1734016143.886156:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c2da0 (ffff88012db7c998/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.886159:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c2da0 (ffff88012db7c998/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.886165:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40ab70 (ffff88012db7c1b8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.886170:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40abb0 (ffff88012db7c638/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.886172:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c2da0 (ffff88012db7c998/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.886175:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40ab70 (ffff88012db7c998/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.886179:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40abb0 (ffff88012db7c998/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.886246:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016143.886275:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7730817 00000080:00200008:2.0:1734016143.886303:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016143.886313:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016143.886421:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016143.886435:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff8800a6c23600/0x37cab3886ce5d006 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31783dd expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016143.886450:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff8800a6c23600/0x37cab3886ce5d006 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31783dd expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016143.886465:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016143.886467:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d006 00000080:00010000:2.0:1734016143.886472:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d006 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016143.886475:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff8800a6c23600/0x37cab3886ce5d006 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31783dd expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016143.886481:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff8800a6c23600/0x37cab3886ce5d006 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31783dd expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016143.886501:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016143.886509:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016143.886524:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.886526:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2930, osc lock ffff8800aafce000, flags 0x400001002 00000008:00010000:2.0:1734016143.886531:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.886532:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2970, osc lock ffff8800aafce260, flags 0x400001002 00010000:00010000:2.0:1734016143.886552:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d07d lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.886570:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d07d lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.886640:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5d084 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.886647:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf840/0x37cab3886ce5d084 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.886861:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332512896:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.886953:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016143.886959:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332512768:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.890404:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016143.895078:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016143.910136:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7730817 i_blocks 216 00000020:00010000:2.0:1734016143.910148:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.910154:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.910162:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2930 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.910168:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2970 (ffff88012db7c518/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.910171:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.910174:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2930 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.910181:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2970 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.910206:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016143.910288:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016143.910297:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016143.910311:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016143.910338:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.910342:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2930, osc lock ffff8800aafce260, flags 0x400001002 00000008:00010000:2.0:1734016143.910348:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.910350:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2970, osc lock ffff8800aafce000, flags 0x400001002 00010000:00010000:2.0:1734016143.910383:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d08b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.910393:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d08b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.910500:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d092 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.910508:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d092 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.910570:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332513152:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.910693:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016143.910701:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332513024:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.931075:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334631040 00000100:00100000:2.0:1734016143.931086:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334631040 00000100:00100000:2.0:1734016143.931148:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14659 00000100:00100000:2.0:1734016143.931159:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334631040:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016143.931172:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.931183:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.931213:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 865, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016143.931237:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 168 00000020:00010000:2.0:1734016143.931242:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016143.931243:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016143 atime: 1734016143 ctime: 1734016143 blocks: 168 00000100:00100000:2.0:1734016143.931303:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334631040:12345-192.168.201.152@tcp:106: Request processed in 145us (980us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.931316:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14659 00010000:00010000:2.0:1734016143.944414:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d08b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.944434:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d08b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.944446:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d08b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.944451:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d08b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.944457:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d08b lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.944470:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016143.944483:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d08b lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.944489:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d08b lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.944495:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d08b lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.944501:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbf600 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016143.944508:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d08b lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.944518:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016143.944523:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d08b lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.944529:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d08b lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.945377:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332513024:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.951957:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334631168 00000100:00100000:2.0:1734016143.951967:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334631168 00000100:00100000:2.0:1734016143.952060:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14660 00000100:00100000:2.0:1734016143.952068:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341d880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:f1fa65c5-ac21-42fc-9fc8-6a110334833e+5:7713:x1818247334631168:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016143.952087:0:4729:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff8800a6c23600/0x37cab3886ce5d006 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb46129d8d31783dd expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016143.953178:0:4729:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016143.953189:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341d880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:f1fa65c5-ac21-42fc-9fc8-6a110334833e+5:7713:x1818247334631168:12345-192.168.201.152@tcp:104: Request processed in 1120us (1371us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016143.953196:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14660 00000080:00200000:2.0:1734016143.968003:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7347115 i_blocks 192 00000020:00010000:2.0:1734016143.968020:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8128 (ffff8800a72d9c68/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.968026:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8128 (ffff8800a72d9c68/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.968034:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86764b0 (ffff88012db7cb48/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.968053:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.968076:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.968102:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86764f0 (ffff88012db7c518/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016143.968108:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.968115:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016143.968124:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8128 (ffff8800a72d9c68/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.968125:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86764b0 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.968130:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86764f0 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.968157:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016143.968179:0:10366:0:(llite_lib.c:2534:ll_setattr_raw()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8) current size 7347115, valid attrs 2168, mode 8800, uid -1451891104, gid -30720, new size 7347115, atime 1734016143.000000000, mtime 1734016143.944000000, ctime 1734016143.944000000, rc 0 00000080:00200008:2.0:1734016143.968211:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016143.968233:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016143.968261:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016143.968277:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6fb6400 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:2.0:1734016143.968333:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016143.968340:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016143.968356:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00000100:00100000:2.0:1734016143.968385:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332513920:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.968509:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2f00 going to sleep for 11 seconds 00000100:00100000:2.0:1734016143.975505:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332513920:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016143.975540:0:10366:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016143.975551:0:10366:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016143.975573:0:10366:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016143.975584:0:10366:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016143.975595:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016143.975600:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016143.975606:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1734016143.975616:0:10366:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a73c9f80 x1818247332513920/t0(0) o101->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016159 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:2.0:1734016143.975627:0:10366:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1734016143.975639:0:10366:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016143.975650:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6fb6400 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:2.0:1734016143.975736:0:10366:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1734016143.975742:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d099 00000080:00010000:2.0:1734016143.975747:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016143.975750:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d099 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016143.975753:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016143.975760:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016143.975808:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016143.975824:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016143.975862:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.975866:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f30, osc lock ffff8800aafce130, flags 0x400001002 00000008:00010000:2.0:1734016143.975876:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.975879:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f70, osc lock ffff8800aafcf430, flags 0x400001002 00010000:00010000:2.0:1734016143.975902:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.975917:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.975939:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.975946:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.975966:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.975971:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.975978:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.975985:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016143.976007:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7347115 i_blocks 192 00000020:00010000:2.0:1734016143.976015:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.976019:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.976022:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f30 (ffff88012db7c518/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.976025:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f70 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.976027:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.976029:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f30 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.976035:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f70 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.976053:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016143.976084:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016143.976087:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016143.976090:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016143.976094:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.976095:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f30, osc lock ffff8800aafcf430, flags 0x400001002 00000008:00010000:2.0:1734016143.976099:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.976100:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f70, osc lock ffff8800aafce130, flags 0x400001002 00010000:00010000:2.0:1734016143.976106:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.976112:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.976118:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.976122:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.976130:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.976137:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.976142:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.976152:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016143.976159:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7347115 i_blocks 192 00000020:00010000:2.0:1734016143.976161:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_release():233 00000020:00010000:2.0:1734016143.976162:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.976164:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f30 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.976166:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f70 (ffff88012db7c518/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016143.976167:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.976169:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f30 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016143.976171:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f70 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016143.976175:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016143.976180:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7347115 00000080:00200008:2.0:1734016143.976196:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016143.976204:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016143.976248:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016143.976259:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b62b8800 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:2.0:1734016143.976307:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d0a0 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016143.976312:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d0a0 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016143.976319:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d0a0 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00000100:00100000:2.0:1734016143.976343:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332514048:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016143.976421:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2f00 going to sleep for 11 seconds 00000100:00100000:2.0:1734016143.981750:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332514048:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016143.981799:0:10366:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d0a0 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d317848c expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016143.981812:0:10366:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d0a0 lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d317848c expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016143.981829:0:10366:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d0a0 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d317848c expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016143.981840:0:10366:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d0a0 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d317848c expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016143.981855:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d0a0 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d317848c expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016143.981863:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d0a0 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d317848c expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016143.981869:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d0a0 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d317848c expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1734016143.981882:0:10366:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a73c9f80 x1818247332514048/t0(0) o101->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016159 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:2.0:1734016143.981893:0:10366:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d0a0 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d317848c expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1734016143.981904:0:10366:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d0a0 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d317848c expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016143.981921:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b62b8800 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:2.0:1734016143.982014:0:10366:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1734016143.982020:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d0a0 00000080:00010000:2.0:1734016143.982027:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016143.982031:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d0a0 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016143.982034:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d0a0 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d317848c expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016143.982041:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d0a0 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d317848c expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016143.982085:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016143.982096:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016143.982115:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.982118:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f30, osc lock ffff8800aafce130, flags 0x400001002 00000008:00010000:2.0:1734016143.982125:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016143.982126:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f70, osc lock ffff8800aafcf430, flags 0x400001002 00010000:00010000:2.0:1734016143.982153:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d0a7 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.982161:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d0a7 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.984148:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d0ae lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.984163:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d0ae lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.984245:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332514304:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016143.990346:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d0ae lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.990360:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d0ae lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.990371:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d0ae lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.990377:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d0ae lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.990383:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d0ae lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.990397:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3152811, setting rss=3152811; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016143.990404:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d0ae lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.990409:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d0ae lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.990415:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d0ae lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.990426:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbc6c0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016143.990431:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d0ae lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.990440:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016143.990444:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d0ae lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.990450:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d0ae lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.991839:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332514304:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016143.992456:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d0a7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.992470:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d0a7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.992482:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d0a7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.992487:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d0a7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.992493:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d0a7 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016143.992518:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016143.992529:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d0a7 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.992535:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d0a7 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.992541:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d0a7 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.992546:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbd8c0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016143.992550:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d0a7 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.992566:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016143.992572:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d0a7 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016143.992578:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d0a7 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016143.992596:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332514176:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016144.026803:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016144.031607:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016144.044268:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7347115 i_blocks 192 00000020:00010000:2.0:1734016144.044277:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.044280:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.044285:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40a030 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.044289:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40a070 (ffff88012db7c518/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.044292:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.044293:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40a030 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.044297:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40a070 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.044306:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.044317:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7347115 00000080:00200008:2.0:1734016144.044336:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016144.044341:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016144.044369:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016144.044388:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.044400:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016144.044414:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016144.044417:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d099 00000080:00010000:2.0:1734016144.044425:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d099 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016144.044428:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.044437:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016144.044461:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.044474:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.044494:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.044497:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e20f0, osc lock ffff8800aafcf1d0, flags 0x400001002 00000008:00010000:2.0:1734016144.044503:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.044505:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2130, osc lock ffff8800aafce850, flags 0x400001002 00010000:00010000:2.0:1734016144.044524:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044531:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044540:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044545:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044573:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044580:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044588:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044594:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.044611:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7347115 i_blocks 192 00000020:00010000:2.0:1734016144.044614:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.044617:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.044620:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e20f0 (ffff88012db7c518/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.044623:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2130 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.044626:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.044627:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e20f0 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.044631:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2130 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.044639:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.044653:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016144.044655:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.044658:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.044662:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.044663:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e20f0, osc lock ffff8800aafce850, flags 0x400001002 00000008:00010000:2.0:1734016144.044665:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.044667:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2130, osc lock ffff8800aafcf1d0, flags 0x400001002 00010000:00010000:2.0:1734016144.044672:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044678:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044686:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044692:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044703:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044709:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044715:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044719:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.044725:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7347115 i_blocks 192 00000020:00010000:2.0:1734016144.044727:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.044729:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.044730:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e20f0 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.044732:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2130 (ffff88012db7c518/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.044734:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.044735:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e20f0 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.044737:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2130 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.044742:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.044746:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7347115 00000080:00200008:2.0:1734016144.044750:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016144.044753:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016144.044759:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016144.044766:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.044773:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016144.044779:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016144.044781:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d099 00000080:00010000:2.0:1734016144.044784:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d099 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016144.044786:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.044793:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016144.044803:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.044806:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.044810:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.044811:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e20f0, osc lock ffff8800aafcf1d0, flags 0x400001002 00000008:00010000:2.0:1734016144.044814:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.044815:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2130, osc lock ffff8800aafce850, flags 0x400001002 00010000:00010000:2.0:1734016144.044820:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044827:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044835:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044845:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044853:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044860:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044866:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044871:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.044881:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7347115 i_blocks 192 00000020:00010000:2.0:1734016144.044883:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.044884:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.044886:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e20f0 (ffff88012db7c518/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.044888:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2130 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.044890:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.044892:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e20f0 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.044894:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2130 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.044899:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.044905:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016144.044907:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.044910:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.044914:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.044916:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e20f0, osc lock ffff8800aafce850, flags 0x400001002 00000008:00010000:2.0:1734016144.044918:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.044920:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2130, osc lock ffff8800aafcf1d0, flags 0x400001002 00010000:00010000:2.0:1734016144.044924:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044931:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044937:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044943:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044951:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044956:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044961:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.044967:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.044979:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7347115 i_blocks 192 00000020:00010000:2.0:1734016144.044981:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.044983:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.044985:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e20f0 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.044987:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2130 (ffff88012db7c518/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.044989:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70de910 (ffff8800a72d9c68/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.044991:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e20f0 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.044994:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2130 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.044999:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.045003:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7347115 00000080:00200008:2.0:1734016144.045143:0:10366:0:(llite_lib.c:2284:ll_setattr_raw()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8) current size 7347115, valid attrs 2068, mode 8800, uid -1451891104, gid -30720, new size 7562523, atime 1734016144.000000000, mtime 1734016144.000000000, ctime 1734016144.000000000 00010000:00010000:2.0:1734016144.045158:0:10366:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.045172:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a72f1d40/0x37cab3886ce591ae lrc: 2/0,0 mode: CR/CR res: [0x200000401:0xc:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3173bb4 expref: -99 pid: 10365 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.045255:0:10366:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 1/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.045261:0:10366:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1734016144.045263:0:10366:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d099 lrc: 0/0,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb46129d8d317847e expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1734016144.045291:0:10366:0:(genops.c:2298:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8800b5805800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1734016144.045303:0:10366:0:(client.c:782:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8800a73ca680 x1818247332515456/t0(0) o36->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734016144.045333:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332515456:192.168.201.152@tcp:36:fsx.0 00000100:00100000:2.0:1734016144.045434:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e20c0 going to sleep for 11 seconds 00000100:00100000:2.0:1734016144.052833:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016144.062296:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332515456:192.168.201.152@tcp:36:fsx.0 00000080:00200000:2.0:1734016144.062351:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:2.0:1734016144.062358:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:2.0:1734016144.062373:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016144.062395:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1734016144.062409:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016144.062414:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e20f0, osc lock ffff8800aafcf1d0, flags 0x0 00000008:00010000:2.0:1734016144.062417:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016144.062419:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2130, osc lock ffff8800aafce850, flags 0x0 00010000:00010000:2.0:1734016144.062440:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.062453:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.062465:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.062472:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.062478:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.062497:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.062505:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.062511:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.062517:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.062522:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.062612:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332515712:192.168.201.152@tcp:10:fsx.0 00000100:00100000:2.0:1734016144.068134:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800b341d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332515840:192.168.201.152@tcp:10:fsx.0 00000100:00100000:2.0:1734016144.073673:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332515840:192.168.201.152@tcp:10:fsx.0 00000020:00010000:2.0:1734016144.085736:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8128 (ffff8800a72d9c68/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.085740:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8128 (ffff8800a72d9c68/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.085745:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f80b0 (ffff88012db7c518/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016144.085755:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.085775:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.085794:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8128 (ffff8800a72d9c68/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.085796:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f80b0 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.085816:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016144.085839:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016144.085855:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016144.085872:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1844, 1844] 00000008:00010000:2.0:1734016144.085890:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016144.085896:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8db0, osc lock ffff8800aafce4c0, flags 0x0 00010000:00010000:2.0:1734016144.085921:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.085935:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3358720 3362815) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.085945:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.085950:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.085955:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.085995:0:10366:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800aa7e2300 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.086010:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af698248, count 1, dirtied 1 00000020:00010000:2.0:1734016144.086021:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8128 (ffff8800a72d9c68/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.086022:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8128 (ffff8800a72d9c68/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.086024:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8db0 (ffff88012db7c518/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016144.086027:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.086032:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.086039:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8128 (ffff8800a72d9c68/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.086040:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8db0 (ffff8800a72d9c68/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.086047:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016144.086088:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 1845 00000080:00200000:2.0:1734016144.086093:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016144.086096:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016144.086100:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1845, 1845] 00000008:00010000:2.0:1734016144.086103:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016144.086105:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8db0, osc lock ffff8800aafce4c0, flags 0x0 00010000:00010000:2.0:1734016144.086109:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.086116:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3362816 3366911) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.086127:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.086132:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.086137:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.086195:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:2.0:1734016144.086204:0:1970:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e26c0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.087828:0:1970:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3362816 3366911) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016144.087872:0:1970:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73c9880 - object 0x240000400:7 offset 3362816<>3366912 00000100:00100000:2.0:1734016144.088135:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332516352:192.168.201.152@tcp:3:fsx.0 00000008:00010000:2.0:1734016144.093325:0:1970:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e26c0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.093370:0:1970:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e26c0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.093390:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332516352:192.168.201.152@tcp:3:fsx.0 00000008:00010000:2.0:1734016144.097791:0:1970:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800a708c840 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.097836:0:1970:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a708c840 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.097856:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332516608:192.168.201.152@tcp:3:fsx.0 00000100:00100000:2.0:1734016144.098959:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800b341d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332516736:192.168.201.152@tcp:4:fsx.0 00000008:00010000:2.0:1734016144.102729:0:1971:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e2300 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.102794:0:1971:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2300 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.102817:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332516736:192.168.201.152@tcp:4:fsx.0 00000020:00010000:2.0:1734016144.131289:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.131293:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.131296:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f87b0 (ffff88012db7c248/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016144.131302:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 4/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.131311:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.131327:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.131329:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f87b0 (ffff88012db7cb48/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.131343:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016144.131389:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 562 00000080:00200000:2.0:1734016144.131397:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016144.131409:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016144.131420:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [562, 562] 00000008:00010000:2.0:1734016144.131430:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016144.131433:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8830, osc lock ffff8800aafcef70, flags 0x0 00010000:00010000:2.0:1734016144.131451:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.131462:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2301952 2306047) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.131470:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 6/1,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.131475:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 6/1,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.131480:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 5/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.131681:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00010000:2.0:1734016144.131690:0:1971:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e2240 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 4/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.131711:0:1971:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2301952 2306047) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 5/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016144.131749:0:1971:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73c9880 - object 0x280000400:7 offset 2301952<>2306048 00000100:00100000:2.0:1734016144.131768:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332517632:192.168.201.152@tcp:3:fsx.0 00000008:00010000:2.0:1734016144.143844:0:1971:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e2240 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 4/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.143901:0:1971:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2240 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 4/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.143926:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332517632:192.168.201.152@tcp:3:fsx.0 00000100:00100000:2.0:1734016144.152859:0:4728:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016144.152881:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334632576 00000100:00100000:2.0:1734016144.152893:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334632576 00000100:00100000:2.0:1734016144.152934:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334632448 00000100:00100000:2.0:1734016144.152935:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334632448 00000100:00100000:2.0:1734016144.152948:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14670 00000100:00100000:2.0:1734016144.152956:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ca80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334632576:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016144.152978:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.152994:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.153032:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 776, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016144.153036:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 3368219 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 48 00000020:00010000:2.0:1734016144.153039:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016144.153041:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 3368219 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 48 00000100:00100000:2.0:1734016144.153098:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ca80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334632576:12345-192.168.201.152@tcp:106: Request processed in 142us (482us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016144.153106:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14670 00000100:00100000:2.0:1734016144.153127:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14671 00000100:00100000:2.0:1734016144.153131:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012b52f050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334632448:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016144.153139:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.153145:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.153156:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 776, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016144.153157:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 168 00000020:00010000:2.0:1734016144.153159:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016144.153161:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 168 00000100:00100000:2.0:1734016144.153171:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012b52f050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334632448:12345-192.168.201.152@tcp:106: Request processed in 39us (425us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016144.153176:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14671 00000080:00200000:2.0:1734016144.158184:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 216 00000020:00010000:2.0:1734016144.158193:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.158196:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.158202:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86763f0 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.158206:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676430 (ffff88012db7c008/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.158208:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.158210:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86763f0 (ffff88012db7cb48/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.158352:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676430 (ffff88012db7cb48/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.158374:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016144.158503:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 430 00000080:00200000:2.0:1734016144.158514:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016144.158530:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016144.158546:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [430, 430] 00000008:00010000:2.0:1734016144.158577:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016144.158582:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8cb0, osc lock ffff8800aafceab0, flags 0x0 00010000:00010000:2.0:1734016144.158612:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.158620:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1761280->1765375] (req 1761280->1765375) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.158667:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1761280->1765375] (req 1761280->1765375) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.158697:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332518016:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016144.159359:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e26c0 going to sleep for 11 seconds 00000100:00100000:2.0:1734016144.162702:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334632704 00000100:00100000:2.0:1734016144.162718:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334632704 00000100:00100000:2.0:1734016144.162795:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14672 00000100:00100000:2.0:1734016144.162805:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334632704:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016144.162830:0:9532:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a6c22880/0x37cab3886ce5d05a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 434176->446463) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d3178431 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.163651:0:9532:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016144.163662:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334632704:12345-192.168.201.152@tcp:104: Request processed in 857us (1300us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016144.163669:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14672 00000100:00100000:2.0:1734016144.168100:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332518016:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016144.168153:0:10366:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1761280->1765375] (req 1761280->1765375) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.168167:0:10366:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1761280->1765375] (req 1761280->1765375) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.168177:0:10366:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1761280->1765375] (req 1761280->1765375) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.168477:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334632832 00000100:00100000:2.0:1734016144.168486:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334632832 00000100:00100000:2.0:1734016144.168548:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14673 00000100:00100000:2.0:1734016144.168563:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7025:x1818247334632832:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016144.168573:0:9532:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1761280->1765375] (req 1761280->1765375) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.168584:0:9532:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1761280->1765375] (req 1761280->1765375) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.168593:0:9532:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.168603:0:9532:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.168612:0:9532:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 6/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.168623:0:9532:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 6/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.168666:0:9532:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff88012dbbd440) 00000100:00100000:2.0:1734016144.168732:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7025:x1818247334632832:12345-192.168.201.152@tcp:105: Request processed in 168us (339us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016144.168741:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14673 00000020:00010000:2.0:1734016144.177557:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.177561:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.177565:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8cb0 (ffff88012db7c008/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016144.177655:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.177673:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.177702:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.177705:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8cb0 (ffff88012db7cb48/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.177740:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016144.178353:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 431 00000080:00200000:2.0:1734016144.178366:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016144.178384:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016144.178403:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [431, 431] 00000008:00010000:2.0:1734016144.178419:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016144.178424:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8cb0, osc lock ffff8800aafceab0, flags 0x0 00010000:00010000:2.0:1734016144.178451:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.178464:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1765376 1769471) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.178474:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.178481:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.178488:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.178592:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00010000:2.0:1734016144.178604:0:1971:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e26c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.178627:0:1971:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1765376 1769471) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016144.178674:0:1971:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73cbb80 - object 0x280000400:7 offset 1765376<>1769472 00000100:00100000:2.0:1734016144.178703:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332518656:192.168.201.152@tcp:3:fsx.0 00000008:00010000:2.0:1734016144.182119:0:1971:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e26c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.182174:0:1971:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e26c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.182202:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332518656:192.168.201.152@tcp:3:fsx.0 00000020:00010000:2.0:1734016144.186490:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.186494:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.186497:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f8430 (ffff88012db7c008/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016144.186506:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.186515:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.186530:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.186533:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f8430 (ffff88012db7cb48/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.186563:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.186754:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016144.186762:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016144.186787:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016144.186806:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.186818:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016144.186835:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016144.186839:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d0df 00000080:00010000:2.0:1734016144.186846:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d0df from it ffff8800a7343cd8 00010000:00010000:2.0:1734016144.186850:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.186856:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016144.186880:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.186897:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.186922:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.186927:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e26f0, osc lock ffff8800aafce5f0, flags 0x400001002 00000008:00010000:2.0:1734016144.186934:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.186935:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2730, osc lock ffff8800aafcf7c0, flags 0x400001002 00010000:00010000:2.0:1734016144.186970:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.186978:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.187074:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.187088:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016144.187105:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332519040:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016144.187242:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.187257:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.187270:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.190900:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.190932:0:1971:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.190949:0:1971:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.190963:0:1971:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.190983:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.191083:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.191093:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.191100:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.191112:0:1971:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.191138:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332519040:192.168.201.152@tcp:101:fsx.0 00000080:00200000:2.0:1734016144.210765:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016144.210784:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016144.210796:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1734016144.221994:0:1971:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8800b5805800: committing for last_committed 4294968266 gen 2 00000100:00100000:2.0:1734016144.222097:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332519296:192.168.201.152@tcp:16:fsx.0 00000100:00100000:2.0:1734016144.237281:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334632960 00000100:00100000:2.0:1734016144.237302:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334632960 00000100:00100000:2.0:1734016144.237384:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14674 00000100:00100000:2.0:1734016144.237399:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334632960:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016144.237428:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.237441:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.237482:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 774, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016144.237484:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 3368219 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 72 00000020:00010000:2.0:1734016144.237487:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016144.237488:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 3368219 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 72 00000100:00100000:2.0:1734016144.237577:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334632960:12345-192.168.201.152@tcp:106: Request processed in 182us (541us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016144.237587:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14674 00000080:00200000:2.0:1734016144.257080:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 240 00000020:00010000:2.0:1734016144.257088:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.257092:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.257098:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86766f0 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.257102:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676730 (ffff88012db7c008/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.257104:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.257106:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86766f0 (ffff88012db7cb48/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.257110:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676730 (ffff88012db7cb48/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.257124:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016144.257196:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 439 00000080:00200000:2.0:1734016144.257207:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016144.257257:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016144.257270:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [439, 439] 00000008:00010000:2.0:1734016144.257285:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016144.257289:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef82b0, osc lock ffff8800aafce720, flags 0x0 00010000:00010000:2.0:1734016144.257307:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.257319:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1798144 1802239) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.257327:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.257333:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.257343:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.257411:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00010000:2.0:1734016144.257420:0:1971:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.257445:0:1971:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1798144 1802239) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016144.257485:0:1971:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73cbb80 - object 0x280000400:7 offset 1798144<>1802240 00000100:00100000:2.0:1734016144.257506:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332519680:192.168.201.152@tcp:3:fsx.0 00000008:00010000:2.0:1734016144.263409:0:1971:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.263443:0:1971:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.263463:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332519680:192.168.201.152@tcp:3:fsx.0 00000020:00010000:2.0:1734016144.276737:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.276741:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.276746:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f8330 (ffff88012db7c008/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016144.276755:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.276770:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.276792:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.276793:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f8330 (ffff88012db7cb48/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.276811:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.277159:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016144.277165:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016144.277184:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016144.277197:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.277207:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016144.277255:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016144.277258:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d0df 00000080:00010000:2.0:1734016144.277264:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d0df from it ffff8800a7343cd8 00010000:00010000:2.0:1734016144.277267:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.277273:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016144.277292:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.277303:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.277322:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.277325:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e29f0, osc lock ffff8800aafcf560, flags 0x400001002 00000008:00010000:2.0:1734016144.277331:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.277333:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2a30, osc lock ffff8800aafcfc80, flags 0x400001002 00010000:00010000:2.0:1734016144.277346:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277353:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277365:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277371:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277387:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277392:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277399:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277406:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.277423:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 240 00000020:00010000:2.0:1734016144.277427:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.277428:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.277430:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e29f0 (ffff88012db7c008/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.277433:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2a30 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.277435:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.277437:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e29f0 (ffff88012db7cb48/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.277441:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2a30 (ffff88012db7cb48/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.277447:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.277463:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016144.277467:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.277471:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.277475:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.277477:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e29f0, osc lock ffff8800aafcfc80, flags 0x400001002 00000008:00010000:2.0:1734016144.277481:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.277482:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2a30, osc lock ffff8800aafcf560, flags 0x400001002 00010000:00010000:2.0:1734016144.277487:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277494:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277501:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277507:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277516:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277522:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277528:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277534:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.277542:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 240 00000020:00010000:2.0:1734016144.277544:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.277547:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.277559:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e29f0 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.277561:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2a30 (ffff88012db7c008/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.277562:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.277564:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e29f0 (ffff88012db7cb48/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.277567:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2a30 (ffff88012db7cb48/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.277571:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.277578:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7562523 00000080:00200008:2.0:1734016144.277582:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016144.277585:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016144.277590:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016144.277596:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.277605:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016144.277611:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016144.277613:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d0df 00000080:00010000:2.0:1734016144.277616:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d0df from it ffff8800a7343cd8 00010000:00010000:2.0:1734016144.277618:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.277623:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016144.277632:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.277636:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.277640:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.277642:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e29f0, osc lock ffff8800aafcf560, flags 0x400001002 00000008:00010000:2.0:1734016144.277644:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.277646:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2a30, osc lock ffff8800aafcfc80, flags 0x400001002 00010000:00010000:2.0:1734016144.277651:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277657:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277663:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277672:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277680:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277685:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277691:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277697:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.277704:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 240 00000020:00010000:2.0:1734016144.277711:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.277713:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.277715:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e29f0 (ffff88012db7c008/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.277717:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2a30 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.277719:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.277721:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e29f0 (ffff88012db7cb48/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.277723:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2a30 (ffff88012db7cb48/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.277727:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.277733:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016144.277737:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.277740:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.277744:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.277746:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e29f0, osc lock ffff8800aafcfc80, flags 0x400001002 00000008:00010000:2.0:1734016144.277748:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.277749:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2a30, osc lock ffff8800aafcf560, flags 0x400001002 00010000:00010000:2.0:1734016144.277754:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277760:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277766:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277772:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277779:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277786:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277791:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277795:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.277805:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 240 00000020:00010000:2.0:1734016144.277806:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.277808:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.277810:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e29f0 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.277812:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2a30 (ffff88012db7c008/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.277814:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.277815:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e29f0 (ffff88012db7cb48/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.277817:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2a30 (ffff88012db7cb48/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.277820:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.277823:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7562523 00000080:00200008:2.0:1734016144.277827:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016144.277830:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016144.277835:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016144.277839:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.277845:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016144.277851:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016144.277852:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d0df 00000080:00010000:2.0:1734016144.277854:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d0df from it ffff8800a7343cd8 00010000:00010000:2.0:1734016144.277857:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.277862:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016144.277871:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.277874:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.277878:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.277880:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e29f0, osc lock ffff8800aafcf560, flags 0x400001002 00000008:00010000:2.0:1734016144.277883:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.277884:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2a30, osc lock ffff8800aafcfc80, flags 0x400001002 00010000:00010000:2.0:1734016144.277889:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277895:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277901:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277906:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277914:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277919:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277925:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277934:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.277941:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 240 00000020:00010000:2.0:1734016144.277943:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.277944:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.277945:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e29f0 (ffff88012db7c008/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.277947:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2a30 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.277949:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.277952:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e29f0 (ffff88012db7cb48/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.277954:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2a30 (ffff88012db7cb48/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.277958:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.277962:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016144.277965:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.277967:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.277971:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.277972:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e29f0, osc lock ffff8800aafcfc80, flags 0x400001002 00000008:00010000:2.0:1734016144.277975:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.277976:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2a30, osc lock ffff8800aafcf560, flags 0x400001002 00010000:00010000:2.0:1734016144.277980:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277987:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.277993:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.278000:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.278007:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.278011:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.278017:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.278022:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.278028:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 240 00000020:00010000:2.0:1734016144.278030:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.278031:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.278033:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e29f0 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.278034:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2a30 (ffff88012db7c008/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.278036:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c0db0 (ffff88012db7cb48/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.278037:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e29f0 (ffff88012db7cb48/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.278038:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2a30 (ffff88012db7cb48/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.278042:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.278045:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7562523 00000080:00a00000:2.0:1734016144.278147:0:10366:0:(llite_mmap.c:611:ll_file_mmap()) VFS_Op: fid=[0x200000401:0xc:0x0] vma=ffff8800a91820d8 start=0x7fef68fb6000 end=0x7fef68fb9000 vm_flags=0xfb 00000080:00200000:2.0:1734016144.278158:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.278163:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.278169:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.278170:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e29f0, osc lock ffff8800aafcf560, flags 0x400001002 00000008:00010000:2.0:1734016144.278175:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.278177:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2a30, osc lock ffff8800aafcfc80, flags 0x400001002 00010000:00010000:2.0:1734016144.278184:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.278191:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.278197:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.278204:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d0f4 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1761280->1765375) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31784ee expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.278260:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d109 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.278266:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d109 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.289936:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334633088 00000100:00100000:2.0:1734016144.289952:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334633088 00000100:00100000:2.0:1734016144.290017:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14675 00000100:00100000:2.0:1734016144.290027:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334633088:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016144.290054:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.290067:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.290115:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 774, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016144.290118:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 3368219 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 72 00000020:00010000:2.0:1734016144.290121:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016144.290123:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 3368219 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 72 00000100:00100000:2.0:1734016144.290185:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334633088:12345-192.168.201.152@tcp:106: Request processed in 160us (405us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016144.290192:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14675 00010000:00010000:2.0:1734016144.300326:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d109 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.300350:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d109 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.300364:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d109 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.300371:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d109 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.300379:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d109 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.300402:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3368219, setting rss=3368219; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016144.300416:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d109 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.300422:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d109 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.300428:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d109 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.300434:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbd200 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016144.300440:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d109 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.300450:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016144.300456:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d109 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.300461:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d109 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.300559:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332520064:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016144.303899:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334633216 00000100:00100000:2.0:1734016144.303913:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334633216 00000100:00100000:2.0:1734016144.303980:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14676 00000100:00100000:2.0:1734016144.303989:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334633216:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016144.304006:0:4729:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012b7146c0/0x37cab3886ce5cff1 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3534848->3538943) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d31783c1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.305877:0:4729:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016144.305896:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334633216:12345-192.168.201.152@tcp:104: Request processed in 1899us (2131us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016144.305910:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14676 00000100:00100000:2.0:1734016144.326726:0:4729:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016144.326743:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334633344 00000100:00100000:2.0:1734016144.326749:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334633344 00000100:00100000:2.0:1734016144.326785:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14677 00000100:00100000:2.0:1734016144.326792:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7039c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7025:x1818247334633344:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016144.326801:0:4729:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [1916928->1921023] (req 1916928->1921023) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.326809:0:4729:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [1916928->1921023] (req 1916928->1921023) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.326816:0:4729:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.326822:0:4729:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.326829:0:4729:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.326843:0:4729:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.326860:0:4729:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a6c21f80) 00000100:00100000:2.0:1734016144.326917:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7039c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7025:x1818247334633344:12345-192.168.201.152@tcp:105: Request processed in 123us (285us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016144.326923:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14677 00000020:00010000:2.0:1734016144.383973:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.383977:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.383980:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f8430 (ffff88012db7c248/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016144.383986:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.383994:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.384007:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.384009:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f8430 (ffff88012db7cb48/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.384025:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016144.384038:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016144.384052:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016144.384063:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1492, 1492] 00000008:00010000:2.0:1734016144.384077:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016144.384081:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8cb0, osc lock ffff8800aafcfdb0, flags 0x0 00010000:00010000:2.0:1734016144.384095:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.384109:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1916928 1921023) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.384117:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.384122:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.384127:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.384152:0:10366:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800aa7e2b40 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.384164:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 1, dirtied 1 00000020:00010000:2.0:1734016144.384173:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.384175:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.384176:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8cb0 (ffff88012db7c248/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016144.384179:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.384184:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.384190:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.384192:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8cb0 (ffff88012db7cb48/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.384201:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016144.384242:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 1493 00000080:00200000:2.0:1734016144.384248:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016144.384252:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016144.384255:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1493, 1493] 00000008:00010000:2.0:1734016144.384260:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016144.384262:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8cb0, osc lock ffff8800aafcfdb0, flags 0x0 00010000:00010000:2.0:1734016144.384266:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.384272:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1921024 1925119) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.384278:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.384283:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.384287:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.384301:0:10366:0:(osc_cache.c:543:osc_extent_merge()) ### extent: ffff8800aa7e2d80 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.384306:0:10366:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2d80 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.384316:0:10366:0:(osc_cache.c:560:osc_extent_merge()) ### extent: ffff8800aa7e2b40 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.384322:0:10366:0:(osc_cache.c:771:osc_extent_find()) ### extent: ffff8800aa7e2b40 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.384328:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 1, dirtied 1 00000020:00010000:2.0:1734016144.384332:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.384334:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.384336:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8cb0 (ffff88012db7c248/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016144.384338:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.384345:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.384352:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.384353:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8cb0 (ffff88012db7cb48/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.384358:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016144.384368:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 1494 00000080:00200000:2.0:1734016144.384371:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016144.384374:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016144.384377:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1494, 1494] 00000008:00010000:2.0:1734016144.384381:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016144.384382:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8cb0, osc lock ffff8800aafcfdb0, flags 0x0 00010000:00010000:2.0:1734016144.384386:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.384396:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1925120 1929215) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.384403:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.384409:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.384415:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.384425:0:10366:0:(osc_cache.c:543:osc_extent_merge()) ### extent: ffff8800aa7e2d80 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.384430:0:10366:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2d80 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.384434:0:10366:0:(osc_cache.c:560:osc_extent_merge()) ### extent: ffff8800aa7e2b40 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.384440:0:10366:0:(osc_cache.c:771:osc_extent_find()) ### extent: ffff8800aa7e2b40 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.384447:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 1, dirtied 1 00000020:00010000:2.0:1734016144.384451:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.384454:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.384455:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8cb0 (ffff88012db7c248/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016144.384458:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.384464:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.384472:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724ee48 (ffff88012db7cb48/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.384473:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8cb0 (ffff88012db7cb48/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.384479:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016144.384491:0:10366:0:(file.c:5185:ll_fsync()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8), start 6111232, end 6123519, datasync 1 00000080:00200000:2.0:1734016144.384496:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016144.384499:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1734016144.384506:0:10366:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e2b40 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.384514:0:10366:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800aa7e2b40 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.384621:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1916928 1921023) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d110 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1916928->1921023) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317850a expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016144.384707:0:10366:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73cbb80 - object 0x240000400:7 offset 1916928<>1929216 00000100:00100000:2.0:1734016144.384758:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016144.384775:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332520704:192.168.201.152@tcp:4:fsx.0 00000080:00200000:2.0:1734016144.396873:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016144.396895:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016144.396909:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1734016144.396981:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016144.396997:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332520832:192.168.201.152@tcp:16:fsx.0 00000080:00200000:2.0:1734016144.414414:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 264 00000080:00200000:2.0:1734016144.414435:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.414512:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016144.414521:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016144.414545:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016144.414574:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b62b8800 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:2.0:1734016144.414628:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d117 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016144.414638:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d117 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016144.414648:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d117 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00000100:00100000:2.0:1734016144.414675:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332520960:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016144.414960:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2b40 going to sleep for 11 seconds 00000100:00100000:2.0:1734016144.440946:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016144.451303:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 264 00000020:00010000:2.0:1734016144.451318:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c3d98 (ffff88012db7c008/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.451322:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c3d98 (ffff88012db7c008/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.451329:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676db0 (ffff88012db7ca28/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.451335:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676df0 (ffff88012db7cbd8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.451338:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c3d98 (ffff88012db7c008/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.451342:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676db0 (ffff88012db7c008/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.451350:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676df0 (ffff88012db7c008/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.451392:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016144.451438:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:2.0:1734016144.451444:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:2.0:1734016144.451459:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016144.451475:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1318, 1318] 00000008:00010000:2.0:1734016144.451488:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016144.451491:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef89b0, osc lock ffff8800aafcfdb0, flags 0x0 00010000:00010000:2.0:1734016144.451512:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.451521:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [1204224->1208319] (req 1204224->1208319) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.451564:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [1204224->1208319] (req 1204224->1208319) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.451590:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8700 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332521216:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016144.451681:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2840 going to sleep for 11 seconds 00000100:00100000:2.0:1734016144.468418:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016144.486199:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334633728 00000100:00100000:2.0:1734016144.486212:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334633728 00000100:00100000:2.0:1734016144.486304:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14680 00000100:00100000:2.0:1734016144.486314:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7025:x1818247334633728:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016144.486327:0:9532:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [1204224->1208319] (req 1204224->1208319) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.486338:0:9532:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [1204224->1208319] (req 1204224->1208319) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.486351:0:9532:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.486363:0:9532:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.486374:0:9532:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.486383:0:9532:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.486411:0:9532:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff88012dbbc6c0) 00000100:00100000:2.0:1734016144.486478:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7025:x1818247334633728:12345-192.168.201.152@tcp:105: Request processed in 165us (554us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016144.486484:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14680 00000020:00010000:2.0:1734016144.498512:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724eb78 (ffff88012db7ca28/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.498518:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724eb78 (ffff88012db7ca28/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.498526:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef89b0 (ffff88012db7cb48/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016144.498554:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.498569:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.498587:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724eb78 (ffff88012db7ca28/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.498590:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef89b0 (ffff88012db7ca28/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.498616:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.498717:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016144.498730:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016144.498765:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016144.498796:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.498810:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016144.498829:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016144.498833:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d0df 00000080:00010000:2.0:1734016144.498840:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d0df from it ffff8800a7343cd8 00010000:00010000:2.0:1734016144.498842:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.498847:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016144.498868:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.498886:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.498920:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.498925:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2cf0, osc lock ffff8800aafcfdb0, flags 0x400001002 00000008:00010000:2.0:1734016144.498930:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.498932:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2d30, osc lock ffff8800aafce980, flags 0x400001002 00010000:00010000:2.0:1734016144.498952:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.498965:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.498979:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.498985:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.499001:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.499009:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.499031:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.499039:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.499062:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 280 00000020:00010000:2.0:1734016144.499065:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.499067:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.499068:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2cf0 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.499072:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2d30 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.499074:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.499076:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2cf0 (ffff88012db7ca28/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.499079:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2d30 (ffff88012db7ca28/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.499087:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.499103:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016144.499107:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.499110:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.499113:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.499115:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2cf0, osc lock ffff8800aafce980, flags 0x400001002 00000008:00010000:2.0:1734016144.499118:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.499119:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2d30, osc lock ffff8800aafcfdb0, flags 0x400001002 00010000:00010000:2.0:1734016144.499124:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.499131:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.499137:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.499142:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.499150:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.499155:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.499163:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.499168:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.499176:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 280 00000020:00010000:2.0:1734016144.499179:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.499181:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.499183:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2cf0 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.499185:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2d30 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.499188:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.499190:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2cf0 (ffff88012db7ca28/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.499192:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2d30 (ffff88012db7ca28/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.499196:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.499204:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7562523 00000080:00200000:2.0:1734016144.499270:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:2.0:1734016144.499273:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:2.0:1734016144.499287:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016144.499301:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1334, 1334] 00000008:00010000:2.0:1734016144.499311:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016144.499313:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef89b0, osc lock ffff8800aafcfdb0, flags 0x0 00010000:00010000:2.0:1734016144.499320:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.499328:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1269760 1273855) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.499338:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.499345:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.499351:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.499362:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 280 00000020:00010000:2.0:1734016144.516853:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724eb78 (ffff88012db7ca28/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.516859:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724eb78 (ffff88012db7ca28/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.516865:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef89b0 (ffff88012db7cb48/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016144.516883:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.516899:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.516920:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724eb78 (ffff88012db7ca28/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.516922:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef89b0 (ffff88012db7ca28/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.516952:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.516987:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016144.516997:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016144.517027:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016144.517050:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.517063:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016144.517084:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016144.517088:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d0df 00000080:00010000:2.0:1734016144.517094:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d0df from it ffff8800a7343cd8 00010000:00010000:2.0:1734016144.517096:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.517102:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016144.517126:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.517142:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.517161:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.517164:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2cf0, osc lock ffff8800aafcfdb0, flags 0x400001002 00000008:00010000:2.0:1734016144.517170:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.517171:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2d30, osc lock ffff8800aafce980, flags 0x400001002 00010000:00010000:2.0:1734016144.517186:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517195:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517206:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517239:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517253:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517259:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517265:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517271:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.517288:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 280 00000020:00010000:2.0:1734016144.517292:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.517294:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.517296:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2cf0 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.517298:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2d30 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.517299:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.517301:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2cf0 (ffff88012db7ca28/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.517303:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2d30 (ffff88012db7ca28/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.517312:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.517333:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016144.517337:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.517656:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.517662:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.517664:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2cf0, osc lock ffff8800aafce980, flags 0x400001002 00000008:00010000:2.0:1734016144.517667:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.517668:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2d30, osc lock ffff8800aafcfdb0, flags 0x400001002 00010000:00010000:2.0:1734016144.517674:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517682:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517689:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517694:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517702:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517708:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517714:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517720:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.517728:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 280 00000020:00010000:2.0:1734016144.517731:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.517732:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.517734:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2cf0 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.517736:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2d30 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.517738:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.517739:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2cf0 (ffff88012db7ca28/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.517742:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2d30 (ffff88012db7ca28/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.517747:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.517753:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7562523 00000080:00200008:2.0:1734016144.517763:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016144.517766:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016144.517772:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016144.517777:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.517782:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016144.517788:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016144.517789:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d0df 00000080:00010000:2.0:1734016144.517792:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d0df from it ffff8800a7343cd8 00010000:00010000:2.0:1734016144.517794:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.517800:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016144.517808:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.517811:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.517815:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.517816:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2cf0, osc lock ffff8800aafcfdb0, flags 0x400001002 00000008:00010000:2.0:1734016144.517818:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.517819:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2d30, osc lock ffff8800aafce980, flags 0x400001002 00010000:00010000:2.0:1734016144.517824:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517830:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517837:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517842:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d0fb lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d31784f5 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517849:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517854:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517860:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517870:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.517878:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 280 00000020:00010000:2.0:1734016144.517879:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.517881:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.517883:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2cf0 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.517885:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2d30 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.517887:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.517888:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2cf0 (ffff88012db7ca28/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.517890:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2d30 (ffff88012db7ca28/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.517895:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016144.517907:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:2.0:1734016144.517910:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:2.0:1734016144.517918:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016144.517931:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [455, 456] 00000008:00010000:2.0:1734016144.517939:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016144.517942:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef89b0, osc lock ffff8800aafce980, flags 0x0 00010000:00010000:2.0:1734016144.517947:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6d80000 type 11 mode 2 res 10737419264/7 (1863680 1871871) 00010000:00010000:2.0:1734016144.517957:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d12c lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.517963:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d12c lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [1863680->1871871] (req 1863680->1871871) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.518008:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d12c lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [1863680->1871871] (req 1863680->1871871) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.518044:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332521728:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016144.518148:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1734016144.530189:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016144.532129:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016144.545495:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332521728:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016144.545535:0:10366:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d12c lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1863680->1871871] (req 1863680->1871871) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d317852d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.545563:0:10366:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d12c lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1863680->1871871] (req 1863680->1871871) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d317852d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.545573:0:10366:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d12c lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1863680->1871871] (req 1863680->1871871) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d317852d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.552356:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000020:00010000:2.0:1734016144.567693:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724eb78 (ffff88012db7ca28/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.567697:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724eb78 (ffff88012db7ca28/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.567702:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef89b0 (ffff88012db7c248/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016144.567719:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d12c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1863680->1871871) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317852d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.567735:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d12c lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1863680->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317852d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.567754:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724eb78 (ffff88012db7ca28/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.567758:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef89b0 (ffff88012db7ca28/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.567782:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.567867:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016144.567878:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016144.567908:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016144.567937:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.567953:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016144.567974:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016144.567978:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d0df 00000080:00010000:2.0:1734016144.567986:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d0df from it ffff8800a7343cd8 00010000:00010000:2.0:1734016144.567989:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.567995:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016144.568021:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.568038:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.568063:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.568065:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c30, osc lock ffff8800aafce980, flags 0x400001002 00000008:00010000:2.0:1734016144.568073:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.568075:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c70, osc lock ffff8800aafcfdb0, flags 0x400001002 00010000:00010000:2.0:1734016144.568091:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d12c lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1863680->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317852d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.568108:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d12c lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1863680->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317852d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.568121:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d12c lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1863680->1871871) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317852d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.568127:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d12c lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1863680->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317852d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.568142:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.568148:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.568158:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.568164:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.568186:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 296 00000020:00010000:2.0:1734016144.568189:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.568190:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.568193:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2c30 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.568195:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2c70 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.568199:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.568201:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2c30 (ffff88012db7ca28/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.568204:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2c70 (ffff88012db7ca28/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.568211:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.568255:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016144.568258:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.568261:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.568266:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.568267:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c30, osc lock ffff8800aafcfdb0, flags 0x400001002 00000008:00010000:2.0:1734016144.568270:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.568271:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c70, osc lock ffff8800aafce980, flags 0x400001002 00010000:00010000:2.0:1734016144.568276:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d12c lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1863680->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317852d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.568282:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d12c lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1863680->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317852d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.568289:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d12c lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1863680->1871871) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317852d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.568293:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d12c lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1863680->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317852d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.568302:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.568310:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.568323:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.568329:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.568335:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 296 00000020:00010000:2.0:1734016144.568337:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.568338:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.568339:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2c30 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.568341:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2c70 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.568343:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.568345:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2c30 (ffff88012db7ca28/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.568347:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2c70 (ffff88012db7ca28/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.568351:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.568359:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7562523 00000080:00200008:2.0:1734016144.568405:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 1292099, type: SEEK_SET 00000080:00200008:2.0:1734016144.568414:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 1292099, type: SEEK_SET, rc = 1292099 00000080:00200008:2.0:1734016144.568426:0:10366:0:(file.c:2466:do_file_write_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 1292099, count: 7537 00000080:00200000:2.0:1734016144.568434:0:10366:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: write ppos: 1292099, bytes: 7537 00000080:00200000:2.0:1734016144.568438:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:2.0:1734016144.568445:0:10366:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b62c6a28[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00200000:2.0:1734016144.568448:0:10366:0:(cl_io.c:195:cl_io_rw_init()) io range: 2 [1292099, 1299636) 0 0 00000080:00200000:2.0:1734016144.568451:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016144.568459:0:10366:0:(file.c:1893:ll_file_io_generic()) Range lock [315, 317] 00020000:00200000:2.0:1734016144.568468:0:10366:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 0 chunk: [1292099, 1299636) 1299636, 7537 00020000:00200000:2.0:1734016144.568473:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016144.568489:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [315, 317] 00000008:00010000:2.0:1734016144.568496:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016144.568498:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef89b0, osc lock ffff8800aafce980, flags 0x0 00010000:00010000:2.0:1734016144.568513:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.568521:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1290240->1302527] (req 1290240->1302527) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.568580:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1290240->1302527] (req 1290240->1302527) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.568599:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332522240:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016144.568689:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1734016144.576457:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334634240 00000100:00100000:2.0:1734016144.576466:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334634240 00000100:00100000:2.0:1734016144.576504:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14684 00000100:00100000:2.0:1734016144.576510:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334634240:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016144.576525:0:9532:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d12c lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1863680->1871871) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d317852d expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.578815:0:9532:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016144.578859:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334634240:12345-192.168.201.152@tcp:104: Request processed in 2327us (2913us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016144.578885:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14684 00000100:00100000:2.0:1734016144.580298:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332522240:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016144.580332:0:10366:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1290240->1302527] (req 1290240->1302527) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.580346:0:10366:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1290240->1302527] (req 1290240->1302527) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.580357:0:10366:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1290240->1302527] (req 1290240->1302527) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.581246:0:4736:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016144.581248:0:4736:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016144.581251:0:4736:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1734016144.584998:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334634368 00000100:00100000:2.0:1734016144.585020:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334634368 00000100:00100000:2.0:1734016144.585090:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14685 00000100:00100000:2.0:1734016144.585102:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6472:x1818247334634368:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016144.585126:0:9532:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1290240->1302527] (req 1290240->1302527) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.585140:0:9532:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1290240->1302527] (req 1290240->1302527) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.585152:0:9532:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.585165:0:9532:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.585177:0:9532:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.585189:0:9532:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.585233:0:9532:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff88012dbbd680) 00000100:00100000:2.0:1734016144.585315:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6472:x1818247334634368:12345-192.168.201.152@tcp:105: Request processed in 215us (468us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016144.585323:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14685 00000080:00200000:2.0:1734016144.592056:0:10366:0:(rw26.c:1006:ll_write_end()) pos 1292099, len 2237, copied 2237 00000080:00200000:2.0:1734016144.592071:0:10366:0:(rw26.c:1044:ll_write_end()) page@ffff8800a703cd80[3 ffff8800b62c6aa8 2 1 ffff8801309c12a0] 00000080:00200000:2.0:1734016144.592073:0:10366:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea00029f9040 1fffff0000080d 3:0 ffff8800a703cd80 315 lru 00000080:00200000:2.0:1734016144.592083:0:10366:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800a703cde0 315: 1< 1 - - > 2< 1290240 0 4096 0x0 0x40009 | ffff8800a97f0708 ffff88012b46a9c0 > 3< 0 0 > 4< 0 0 8 489881600 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1734016144.592087:0:10366:0:(rw26.c:1044:ll_write_end()) end page@ffff8800a703cd80 00000080:00200000:2.0:1734016144.592089:0:10366:0:(rw26.c:1044:ll_write_end()) queued page: 1. 00000080:00200000:2.0:1734016144.592097:0:10366:0:(rw26.c:819:ll_write_begin()) Writing 316 of 0 to 4096 bytes 00000080:00200000:2.0:1734016144.592185:0:10366:0:(rw26.c:1006:ll_write_end()) pos 1294336, len 4096, copied 4096 00000080:00200000:2.0:1734016144.592189:0:10366:0:(rw26.c:1044:ll_write_end()) page@ffff88012c776360[3 ffff8800b62c6aa8 2 1 ffff8801309c12a0] 00000080:00200000:2.0:1734016144.592192:0:10366:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002a6f840 1fffff00000805 3:0 ffff88012c776360 316 lru 00000080:00200000:2.0:1734016144.592196:0:10366:0:(rw26.c:1044:ll_write_end()) osc-page@ffff88012c7763c0 316: 1< 0 - - > 2< 1294336 0 0 0x0 0x0 | ffff8800a97f0708 ffff88012b46a9c0 > 3< 0 0 > 4< 0 0 8 489881600 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1734016144.592197:0:10366:0:(rw26.c:1044:ll_write_end()) end page@ffff88012c776360 00000080:00200000:2.0:1734016144.592198:0:10366:0:(rw26.c:1044:ll_write_end()) queued page: 2. 00000080:00200000:2.0:1734016144.592200:0:10366:0:(rw26.c:819:ll_write_begin()) Writing 317 of 0 to 1204 bytes 00000100:00100000:2.0:1734016144.592386:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00010000:2.0:1734016144.592401:0:1971:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e2cc0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.592454:0:1971:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1298432 1302527) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016144.592512:0:1971:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73c8e00 - object 0x280000400:7 offset 1298432<>1302528 00000100:00100000:2.0:1734016144.593318:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332522880:192.168.201.152@tcp:3:fsx.0 00000008:00010000:2.0:1734016144.603066:0:1971:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e2cc0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.603120:0:1971:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2cc0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.603156:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332522880:192.168.201.152@tcp:3:fsx.0 00000100:00100000:2.0:1734016144.607534:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334634496 00000100:00100000:2.0:1734016144.607552:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334634496 00000100:00100000:2.0:1734016144.610348:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14686 00000100:00100000:2.0:1734016144.610369:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334634496:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016144.610396:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.610414:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.610454:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016144.610457:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 3368219 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 112 00000020:00010000:2.0:1734016144.610460:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016144.610461:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 3368219 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 112 00000100:00100000:2.0:1734016144.612990:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334634496:12345-192.168.201.152@tcp:106: Request processed in 2610us (5553us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016144.613006:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14686 00000080:00200000:2.0:1734016144.615933:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 296 00000020:00010000:2.0:1734016144.615940:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.615943:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.615948:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676030 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.615954:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676070 (ffff88012db7c6c8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.615956:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1240 (ffff88012db7ca28/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.615957:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676030 (ffff88012db7ca28/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.615962:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676070 (ffff88012db7ca28/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.615976:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.616030:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016144.616036:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.616044:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.616061:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.616064:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2cf0, osc lock ffff8800aafcfdb0, flags 0x400001002 00000008:00010000:2.0:1734016144.616071:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.616072:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2d30, osc lock ffff8800aafcee40, flags 0x400001002 00010000:00010000:2.0:1734016144.616096:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.616109:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.616118:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.616123:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.616159:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d141 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.616166:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d141 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.620791:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016144.620864:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332523136:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016144.625776:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334634624 00000100:00100000:2.0:1734016144.625795:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334634624 00000100:00100000:2.0:1734016144.625874:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14687 00000100:00100000:2.0:1734016144.625885:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334634624:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016144.625914:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.625928:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.625980:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016144.625983:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 3368219 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 112 00000020:00010000:2.0:1734016144.625986:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016144.625988:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 3368219 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 112 00000100:00100000:2.0:1734016144.626055:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334634624:12345-192.168.201.152@tcp:106: Request processed in 172us (431us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016144.626062:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14687 00010000:00010000:2.0:1734016144.631738:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d141 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.631773:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d141 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.631788:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d141 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.631795:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d141 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.631801:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d141 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.631820:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3368219, setting rss=3368219; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016144.631853:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d141 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.631858:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d141 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.631866:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d141 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.631873:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbf600 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016144.631879:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d141 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.631896:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016144.631905:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d141 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.631912:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d141 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.640725:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332523136:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016144.643748:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334634752 00000100:00100000:2.0:1734016144.643760:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334634752 00000100:00100000:2.0:1734016144.643818:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14688 00000100:00100000:2.0:1734016144.643826:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334634752:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016144.643846:0:9532:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d125 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1204224->1208319) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d3178526 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.643931:0:9532:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016144.643939:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334634752:12345-192.168.201.152@tcp:104: Request processed in 113us (544us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016144.643946:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14688 00000100:00100000:2.0:1734016144.649369:0:4729:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016144.649407:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334634880 00000100:00100000:2.0:1734016144.649417:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334634880 00000100:00100000:2.0:1734016144.649476:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14689 00000100:00100000:2.0:1734016144.649487:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012b52f050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6472:x1818247334634880:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016144.649513:0:4729:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [520192->528383] (req 520192->528383) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.649533:0:4729:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [520192->528383] (req 520192->528383) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.649553:0:4729:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.649570:0:4729:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.649578:0:4729:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.649585:0:4729:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.649616:0:4729:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a6c20b40) 00000100:00100000:2.0:1734016144.649681:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012b52f050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6472:x1818247334634880:12345-192.168.201.152@tcp:105: Request processed in 197us (546us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016144.649689:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14689 00000020:00010000:2.0:1734016144.665906:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724e128 (ffff88012db7c6c8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.665915:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724e128 (ffff88012db7c6c8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.665925:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f8fb0 (ffff88012db7cb48/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016144.665943:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.665969:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.666004:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724e128 (ffff88012db7c6c8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.666008:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f8fb0 (ffff88012db7c6c8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.666034:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.666082:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016144.666096:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016144.666134:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016144.666166:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.666186:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016144.666213:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016144.666491:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d0df 00000080:00010000:2.0:1734016144.666498:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d0df from it ffff8800a7343cd8 00010000:00010000:2.0:1734016144.666503:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.666509:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016144.666537:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.666560:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.666593:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.666596:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2db0, osc lock ffff8800aafced10, flags 0x400001002 00000008:00010000:2.0:1734016144.666605:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.666606:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2df0, osc lock ffff8800aafcf690, flags 0x400001002 00010000:00010000:2.0:1734016144.666646:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d14f lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.666655:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d14f lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.671645:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d156 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.671676:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d156 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.674898:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334635008 00000100:00100000:2.0:1734016144.674918:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334635008 00000100:00100000:2.0:1734016144.674978:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14690 00000100:00100000:2.0:1734016144.674989:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334635008:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016144.675011:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.675026:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.675065:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 4, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016144.675067:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 184 00000020:00010000:2.0:1734016144.675070:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 3, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016144.675072:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 184 00000100:00100000:2.0:1734016144.675146:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334635008:12345-192.168.201.152@tcp:106: Request processed in 159us (888us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016144.675158:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14690 00000100:00100000:2.0:1734016144.681584:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334635136 00000100:00100000:2.0:1734016144.681597:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334635136 00000100:00100000:2.0:1734016144.682375:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14691 00000100:00100000:2.0:1734016144.682387:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334635136:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016144.682410:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.682425:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.682468:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 4, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016144.682471:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 3368219 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 112 00000020:00010000:2.0:1734016144.682475:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016144.682476:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 3368219 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 112 00000100:00100000:2.0:1734016144.683055:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334635136:12345-192.168.201.152@tcp:106: Request processed in 668us (2110us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016144.683063:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14691 00010000:00010000:2.0:1734016144.688970:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d156 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.688993:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d156 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.689010:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d156 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.689022:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d156 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.689029:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d156 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.689048:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 3368219, setting rss=3368219; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016144.689074:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d156 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.689079:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d156 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.689085:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d156 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.689094:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbc000 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016144.689100:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d156 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.689113:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016144.689119:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d156 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.689125:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d156 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.689781:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332523776:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016144.697153:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016144.698319:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016144.710617:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016144.710651:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334635520 00000100:00100000:2.0:1734016144.710660:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334635520 00000100:00100000:2.0:1734016144.710707:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14694 00000100:00100000:2.0:1734016144.710716:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73c8e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334635520:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016144.710731:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.710741:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.710772:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 4, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016144.710774:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 184 00000020:00010000:2.0:1734016144.710777:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 3, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016144.710779:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 184 00000100:00100000:2.0:1734016144.710812:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73c8e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334635520:12345-192.168.201.152@tcp:106: Request processed in 97us (196us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016144.710819:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14694 00000100:00100000:2.0:1734016144.718469:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016144.729897:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016144.729968:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016144.755780:0:10366:0:(rw26.c:1006:ll_write_end()) pos 7092772, len 1500, copied 1500 00000080:00200000:2.0:1734016144.755793:0:10366:0:(rw26.c:1044:ll_write_end()) page@ffff8800a86185e8[3 ffff8800b62c6aa8 2 1 ffff8801309c1978] 00000080:00200000:2.0:1734016144.755796:0:10366:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea00029f8380 1fffff0000080d 3:0 ffff8800a86185e8 1731 lru 00000080:00200000:2.0:1734016144.755804:0:10366:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800a8618648 707: 1< 1 - - > 2< 2895872 0 4096 0x0 0x40009 | ffff8800a97ac758 ffff88012b46ab60 > 3< 0 0 > 4< 0 0 8 273010688 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1734016144.755806:0:10366:0:(rw26.c:1044:ll_write_end()) end page@ffff8800a86185e8 00000080:00200000:2.0:1734016144.755807:0:10366:0:(rw26.c:1044:ll_write_end()) queued page: 1. 00000080:00200000:2.0:1734016144.755811:0:10366:0:(rw26.c:819:ll_write_begin()) Writing 1732 of 0 to 3899 bytes 00000100:00100000:2.0:1734016144.755945:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:2.0:1734016144.755955:0:1970:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e23c0 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.756000:0:1970:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2899968 2904063) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016144.756045:0:1970:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73c8000 - object 0x240000400:7 offset 2899968<>2904064 00000100:00100000:2.0:1734016144.756065:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332525056:192.168.201.152@tcp:3:fsx.0 00000080:00200000:2.0:1734016144.769006:0:10366:0:(rw26.c:1006:ll_write_end()) pos 7094272, len 3899, copied 3899 00000080:00200000:2.0:1734016144.769018:0:10366:0:(rw26.c:1044:ll_write_end()) page@ffff88012c776510[3 ffff8800b62c6aa8 2 1 ffff8801309c1978] 00000080:00200000:2.0:1734016144.769020:0:10366:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002a31400 1fffff0000080d 3:0 ffff88012c776510 1732 lru 00000080:00200000:2.0:1734016144.769030:0:10366:0:(rw26.c:1044:ll_write_end()) osc-page@ffff88012c776570 708: 1< 1 - - > 2< 2899968 0 4096 0x0 0x40009 | ffff8800a97ac758 ffff88012b46ab60 > 3< 0 0 > 4< 0 0 8 273010688 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1734016144.769032:0:10366:0:(rw26.c:1044:ll_write_end()) end page@ffff88012c776510 00000080:00200000:2.0:1734016144.769033:0:10366:0:(rw26.c:1044:ll_write_end()) queued page: 2. 00000080:00200000:2.0:1734016144.769042:0:10366:0:(vvp_io.c:1218:vvp_io_write_commit()) commit async pages: 2, from 2596, to 3899 00000008:00010000:2.0:1734016144.769070:0:10366:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800aa7e2f00 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.769089:0:10366:0:(osc_cache.c:2323:osc_queue_async_io()) ### extent: ffff8800aa7e2f00 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.769102:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 2, dirtied 2 00000080:00200000:2.0:1734016144.769107:0:10366:0:(vvp_io.c:1241:vvp_io_write_commit()) Committed 2 pages 5399 bytes, tot: 5399 00000080:00200000:2.0:1734016144.769110:0:10366:0:(vvp_io.c:1405:vvp_io_write_start()) f16b.sanityn: write bytes 0, result: 5399 00000020:00010000:2.0:1734016144.769129:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724e128 (ffff88012db7c6c8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.769132:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724e128 (ffff88012db7c6c8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.769137:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b352ea30 (ffff88012db7cb48/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016144.769147:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.769158:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.769176:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724e128 (ffff88012db7c6c8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.769178:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b352ea30 (ffff88012db7c6c8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.769187:0:10366:0:(vvp_io.c:533:vvp_io_advance()) advancing 5399 bytes 00000080:00200000:2.0:1734016144.769200:0:10366:0:(file.c:1930:ll_file_io_generic()) Range unlock [1731, 1732] 00000080:00200000:2.0:1734016144.769213:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016144.769239:0:10366:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 2 io complete with rc: 0, result: 5399, restart: 0 00000080:00200000:2.0:1734016144.769256:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 296 00000080:00200000:2.0:1734016144.769270:0:10366:0:(file.c:2044:ll_file_io_generic()) iot: 2, result: 5399 00000080:00200008:2.0:1734016144.769315:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016144.769331:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016144.769376:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016144.769420:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d117 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178518 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.769437:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d117 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178518 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016144.769456:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016144.769461:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d117 00000080:00010000:2.0:1734016144.769471:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d117 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016144.769474:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d117 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178518 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.769480:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d117 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178518 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016144.769509:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.769530:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.769568:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.769573:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2330, osc lock ffff8800aafcfa20, flags 0x400001002 00000008:00010000:2.0:1734016144.769582:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.769583:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2370, osc lock ffff8800aafcfb50, flags 0x400001002 00010000:00010000:2.0:1734016144.769617:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.769628:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.769642:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.769650:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.769674:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.769689:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.769696:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.769703:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.769713:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 296 00000020:00010000:2.0:1734016144.769715:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1918 (ffff88012db7c6c8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.769717:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1918 (ffff88012db7c6c8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.769720:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2330 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.769723:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2370 (ffff88012db7c7e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.769726:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1918 (ffff88012db7c6c8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.769727:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2330 (ffff88012db7c6c8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.769733:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2370 (ffff88012db7c6c8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.769740:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.769757:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016144.769762:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.769764:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.769769:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.769770:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2330, osc lock ffff8800aafcfb50, flags 0x400001002 00000008:00010000:2.0:1734016144.769774:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.769775:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2370, osc lock ffff8800aafcfa20, flags 0x400001002 00010000:00010000:2.0:1734016144.769781:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.769792:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.769800:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.769807:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.769815:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.769820:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.769826:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.769833:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.769843:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 296 00000020:00010000:2.0:1734016144.769845:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1918 (ffff88012db7c6c8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.769847:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1918 (ffff88012db7c6c8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.769849:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2330 (ffff88012db7c7e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.769852:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2370 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.769855:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1918 (ffff88012db7c6c8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.769857:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2330 (ffff88012db7c6c8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.769859:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2370 (ffff88012db7c6c8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.769865:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.769875:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7562523 00000080:00200008:2.0:1734016144.769955:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 6320026, type: SEEK_SET 00000080:00200008:2.0:1734016144.769960:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 6320026, type: SEEK_SET, rc = 6320026 00000080:00200008:2.0:1734016144.769972:0:10366:0:(file.c:2290:do_file_read_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 6320026, count: 841 00000080:00200000:2.0:1734016144.769975:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016144.769981:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016144.770026:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 1542 00000080:00200000:2.0:1734016144.770036:0:10366:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: read ppos: 6320026, bytes: 841 00000080:00200000:2.0:1734016144.770041:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:2.0:1734016144.770049:0:10366:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b62c6a28[0x4, 6, [0x200000401:0xc:0x0] hash] 00000020:00200000:2.0:1734016144.770051:0:10366:0:(cl_io.c:195:cl_io_rw_init()) io range: 1 [6320026, 6320867) 0 0 00000080:00200000:2.0:1734016144.770054:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016144.770075:0:10366:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 1 chunk: [6320026, 6320867) 6320867, 841 00020000:00200000:2.0:1734016144.770082:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016144.770108:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [1542, 1543] 00000008:00010000:2.0:1734016144.770119:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016144.770123:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8830, osc lock ffff8800aafcfa20, flags 0x0 00010000:00010000:2.0:1734016144.770129:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.770139:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2121728 2129919) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.770149:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 7/1,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.770161:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 7/1,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.770168:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 6/1,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.770175:0:10366:0:(vvp_io.c:840:vvp_io_read_start()) f16b.sanityn: read [6320026, 6320867) 00000080:00200000:2.0:1734016144.770190:0:10366:0:(vvp_io.c:899:vvp_io_read_start()) count:841 iocb pos:6320026 00010000:00010000:2.0:1734016144.770271:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.770281:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.770287:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 6/1,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.770291:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 6/1,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.770298:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 5/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.786673:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 7/2,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.786705:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 6/1,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.901762:0:1971:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e2300 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 5/1,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016144.906807:0:1971:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2300 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.906851:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332525312:192.168.201.152@tcp:3:fsx.0 00000080:00200000:2.0:1734016144.923119:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 296 00000020:00010000:2.0:1734016144.923124:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1918 (ffff88012db7c6c8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.923127:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1918 (ffff88012db7c6c8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.923132:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a708c6f0 (ffff88012db7c7e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.923135:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a708c730 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.923138:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1918 (ffff88012db7c6c8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.923139:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a708c6f0 (ffff88012db7c6c8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.923142:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a708c730 (ffff88012db7c6c8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.923153:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016144.923178:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:2.0:1734016144.923182:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:2.0:1734016144.923192:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016144.923201:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1845, 1845] 00000008:00010000:2.0:1734016144.923232:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016144.923236:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8db0, osc lock ffff8800aafcfb50, flags 0x0 00010000:00010000:2.0:1734016144.923255:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.923264:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [3362816->3366911] (req 3362816->3366911) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.923294:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [3362816->3366911] (req 3362816->3366911) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.923315:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332525952:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016144.923399:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2840 going to sleep for 11 seconds 00000100:00100000:2.0:1734016144.927161:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016144.940311:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332525952:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016144.940370:0:10366:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [3362816->3366911] (req 3362816->3366911) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d3178596 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.940388:0:10366:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [3362816->3366911] (req 3362816->3366911) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178596 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.940399:0:10366:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [3362816->3366911] (req 3362816->3366911) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178596 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016144.940949:0:4737:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016144.947711:0:4737:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1734016144.947817:0:4737:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 2/0,0 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.947830:0:4737:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1734016144.947888:0:4737:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.947896:0:4737:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c20b40/0x37cab3886ce5d148 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 520192->528383) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d3178549 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.947919:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016144.947921:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016144.947926:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1734016144.957302:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016144.967523:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332526208:192.168.201.152@tcp:103:ldlm_bl.0 00000020:00010000:2.0:1734016144.972237:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724e128 (ffff88012db7c6c8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.972242:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724e128 (ffff88012db7c6c8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.972249:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8db0 (ffff88012db7cb48/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016144.972263:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3362816->3366911) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178596 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.972281:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3362816->3366911) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178596 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.972300:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724e128 (ffff88012db7c6c8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.972302:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8db0 (ffff88012db7c6c8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.972325:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.972398:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016144.972405:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016144.972429:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016144.972453:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.972464:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016144.972478:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016144.972482:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d0df 00000080:00010000:2.0:1734016144.972487:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d0df from it ffff8800a7343cd8 00010000:00010000:2.0:1734016144.972489:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.972494:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016144.972513:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.972524:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.972551:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.972554:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f30, osc lock ffff8800aafcfb50, flags 0x400001002 00000008:00010000:2.0:1734016144.972558:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.972559:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f70, osc lock ffff8800aafcf8f0, flags 0x400001002 00010000:00010000:2.0:1734016144.972584:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5d19c lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.972592:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5d19c lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.972679:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3362816->3366911) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178596 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.972687:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3362816->3366911) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178596 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.972695:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3362816->3366911) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178596 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.972700:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3362816->3366911) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178596 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.972718:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016144.972733:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332526464:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016144.976623:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016144.981970:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 312 00000020:00010000:2.0:1734016144.981977:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1918 (ffff88012db7c6c8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.981980:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1918 (ffff88012db7c6c8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.981986:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f30 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.981990:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f70 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.981992:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1918 (ffff88012db7c6c8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.981994:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f30 (ffff88012db7c6c8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.981999:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f70 (ffff88012db7c6c8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.982013:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.982033:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016144.982038:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.982043:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.982053:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.982054:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f30, osc lock ffff8800aafcf8f0, flags 0x400001002 00000008:00010000:2.0:1734016144.982058:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.982059:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f70, osc lock ffff8800aafcfb50, flags 0x400001002 00010000:00010000:2.0:1734016144.982079:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d1a3 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.982088:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d1a3 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.982150:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3362816->3366911) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178596 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.982158:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3362816->3366911) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178596 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.982167:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3362816->3366911) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178596 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.982173:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3362816->3366911) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178596 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.984961:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016144.984989:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334636672 00000100:00100000:2.0:1734016144.984996:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334636672 00000100:00100000:2.0:1734016144.985032:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14702 00000100:00100000:2.0:1734016144.985042:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73c8e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334636672:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016144.985058:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.985067:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016144.985098:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016144.985101:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 184 00000020:00010000:2.0:1734016144.985104:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 3, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016144.985106:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016144 atime: 1734016144 ctime: 1734016144 blocks: 184 00000100:00100000:2.0:1734016144.985142:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73c8e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334636672:12345-192.168.201.152@tcp:106: Request processed in 103us (194us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016144.985148:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14702 00000080:00200000:2.0:1734016144.990171:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 312 00000020:00010000:2.0:1734016144.990180:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1918 (ffff88012db7c6c8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.990183:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1918 (ffff88012db7c6c8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.990193:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f30 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.990199:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f70 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.990205:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1918 (ffff88012db7c6c8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.990207:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f30 (ffff88012db7c6c8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.990234:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f70 (ffff88012db7c6c8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.990256:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016144.990277:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 7562523 00000080:00200008:2.0:1734016144.990319:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016144.990327:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016144.990367:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016144.990398:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.990415:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016144.990436:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016144.990440:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d0df 00000080:00010000:2.0:1734016144.990446:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d0df from it ffff8800a7343cd8 00010000:00010000:2.0:1734016144.990449:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016144.990456:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016144.990480:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016144.990495:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016144.990523:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.990526:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f30, osc lock ffff8800aafcfb50, flags 0x400001002 00000008:00010000:2.0:1734016144.990532:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016144.990533:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f70, osc lock ffff8800aafcf8f0, flags 0x400001002 00010000:00010000:2.0:1734016144.990566:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5d1aa lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.990573:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5d1aa lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.990655:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3362816->3366911) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178596 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.990661:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3362816->3366911) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178596 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.990671:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3362816->3366911) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178596 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.990678:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3362816->3366911) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178596 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.993619:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016144.997325:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 7562523 i_blocks 312 00000020:00010000:2.0:1734016144.997334:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1918 (ffff88012db7c6c8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016144.997338:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1918 (ffff88012db7c6c8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.997345:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f30 (ffff88012db7cb48/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.997349:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f70 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016144.997353:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1918 (ffff88012db7c6c8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.997354:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f30 (ffff88012db7c6c8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016144.997361:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f70 (ffff88012db7c6c8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016144.997380:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016144.997412:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:2.0:1734016144.997416:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:2.0:1734016144.997431:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016144.997445:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [309, 310] 00000008:00010000:2.0:1734016144.997457:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016144.997460:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8db0, osc lock ffff8800aafcf8f0, flags 0x0 00010000:00010000:2.0:1734016144.997473:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6d80000 type 11 mode 2 res 10737419264/7 (1265664 1273855) 00010000:00010000:2.0:1734016144.997489:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d1b1 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.997497:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d1b1 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [1265664->1273855] (req 1265664->1273855) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016144.997549:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d1b1 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [1265664->1273855] (req 1265664->1273855) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016144.997580:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332526848:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016144.997652:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2f00 going to sleep for 11 seconds 00000100:00100000:2.0:1734016145.000339:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00010000:00010000:2.0:1734016145.000634:0:4736:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1734016145.007184:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332526848:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016145.007247:0:10366:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d1b1 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1265664->1273855] (req 1265664->1273855) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31785b2 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.007261:0:10366:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d1b1 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1265664->1273855] (req 1265664->1273855) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31785b2 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.007270:0:10366:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d1b1 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1265664->1273855] (req 1265664->1273855) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31785b2 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016145.009268:0:5236:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016145.013912:0:5236:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1734016145.013971:0:5236:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 2/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.013981:0:5236:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1734016145.014756:0:5236:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.014767:0:5236:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d133 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1290240->1302527) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d3178534 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.014796:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016145.014798:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016145.014802:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1734016145.019491:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016145.025526:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332527104:192.168.201.152@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1734016145.044742:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334637184 00000100:00100000:2.0:1734016145.044761:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334637184 00000100:00100000:2.0:1734016145.044852:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14706 00000100:00100000:2.0:1734016145.044862:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:5762f609-0e87-4e8c-9bd6-d528092c9d9c+5:7713:x1818247334637184:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016145.044880:0:4728:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d0df lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb46129d8d31784d9 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016145.044984:0:4728:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016145.044992:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:5762f609-0e87-4e8c-9bd6-d528092c9d9c+5:7713:x1818247334637184:12345-192.168.201.152@tcp:104: Request processed in 132us (563us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.044999:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14706 00000100:00100000:2.0:1734016145.065922:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334637312 00000100:00100000:2.0:1734016145.065936:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334637312 00000100:00100000:2.0:1734016145.066000:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14707 00000100:00100000:2.0:1734016145.066008:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334637312:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016145.066030:0:4729:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d1b1 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1265664->1273855) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d31785b2 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.066102:0:4729:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016145.066109:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334637312:12345-192.168.201.152@tcp:104: Request processed in 100us (341us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.066115:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14707 00000100:00100000:2.0:1734016145.081226:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334637440 00000100:00100000:2.0:1734016145.081241:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334637440 00000100:00100000:2.0:1734016145.081570:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14708 00000100:00100000:2.0:1734016145.081578:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7025:x1818247334637440:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016145.081589:0:4728:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012b714000/0x37cab3886ce5d1b8 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31785c0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.081598:0:4728:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012b714000/0x37cab3886ce5d1b8 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31785c0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.081605:0:4728:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012b714000/0x37cab3886ce5d1b8 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31785c0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.081615:0:4728:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012b714000/0x37cab3886ce5d1b8 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31785c0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.081625:0:4728:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012b714000/0x37cab3886ce5d1b8 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31785c0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.081634:0:4728:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012b714000/0x37cab3886ce5d1b8 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d31785c0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.081653:0:4728:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff88012b714000) 00000100:00100000:2.0:1734016145.081709:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7025:x1818247334637440:12345-192.168.201.152@tcp:105: Request processed in 131us (681us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.081718:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14708 00000100:00100000:2.0:1734016145.092939:0:4729:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016145.092960:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334637568 00000100:00100000:2.0:1734016145.092967:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334637568 00000100:00100000:2.0:1734016145.093011:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14709 00000100:00100000:2.0:1734016145.093019:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012b52d450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334637568:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016145.093038:0:4729:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d195 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3362816->3366911) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d3178596 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.093119:0:4729:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016145.093125:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012b52d450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334637568:12345-192.168.201.152@tcp:104: Request processed in 105us (368us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.093132:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14709 00000100:00100000:2.0:1734016145.104997:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334637696 00000100:00100000:2.0:1734016145.105006:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334637696 00000100:00100000:2.0:1734016145.105046:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14710 00000100:00100000:2.0:1734016145.105052:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7025:x1818247334637696:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016145.105064:0:4728:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.105074:0:4728:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.105084:0:4728:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.105093:0:4728:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.105102:0:4728:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.105109:0:4728:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.105145:0:4728:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff88012b717840) 00000100:00100000:2.0:1734016145.105201:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7025:x1818247334637696:12345-192.168.201.152@tcp:105: Request processed in 149us (445us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.105207:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14710 00000080:00200000:2.0:1734016145.107748:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9360674 i_blocks 352 00000020:00010000:2.0:1734016145.107757:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8218 (ffff8800a72d9d88/0) at cl_lock_release():233 00000020:00010000:2.0:1734016145.107760:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8218 (ffff8800a72d9d88/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.107765:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40af30 (ffff8800a72d9a28/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016145.107775:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012b714000/0x37cab3886ce5d1b8 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31785c0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.107786:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012b714000/0x37cab3886ce5d1b8 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31785c0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.107801:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40af70 (ffff8800a72d91b8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016145.107805:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.107811:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.107819:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8218 (ffff8800a72d9d88/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.107821:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40af30 (ffff8800a72d9d88/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.107825:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40af70 (ffff8800a72d9d88/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016145.107883:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016145.107909:0:10366:0:(llite_lib.c:2534:ll_setattr_raw()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8) current size 9360674, valid attrs 2168, mode 8800, uid -1451891104, gid -30720, new size 9360674, atime 1734016145.000000000, mtime 1734016145.027000000, ctime 1734016145.027000000, rc 0 00000080:00200008:2.0:1734016145.107942:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016145.107953:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016145.107975:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016145.107991:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6fb6400 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:2.0:1734016145.108029:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d1c6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016145.108036:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d1c6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016145.108045:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d1c6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00000100:00100000:2.0:1734016145.108063:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332528512:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016145.108115:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e26c0 going to sleep for 11 seconds 00000100:00100000:2.0:1734016145.111902:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332528512:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016145.111944:0:10366:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d1c6 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d31785d5 expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016145.111957:0:10366:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d1c6 lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d31785d5 expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016145.111972:0:10366:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d1c6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d31785d5 expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016145.111984:0:10366:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d1c6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d31785d5 expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016145.111997:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d1c6 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d31785d5 expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016145.112003:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d1c6 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d31785d5 expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016145.112008:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d1c6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d31785d5 expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1734016145.112017:0:10366:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a73c8000 x1818247332528512/t0(0) o101->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016161 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:2.0:1734016145.112027:0:10366:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d1c6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d31785d5 expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1734016145.112040:0:10366:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d1c6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d31785d5 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016145.112057:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6fb6400 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:2.0:1734016145.112137:0:10366:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1734016145.112144:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d1c6 00000080:00010000:2.0:1734016145.112149:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016145.112154:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d1c6 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016145.112157:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d1c6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31785d5 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016145.112163:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d1c6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31785d5 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016145.112211:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016145.112264:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016145.112289:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016145.112292:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e26f0, osc lock ffff8800aafcfb50, flags 0x400001002 00000008:00010000:2.0:1734016145.112299:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016145.112300:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2730, osc lock ffff8800aafcf0a0, flags 0x400001002 00010000:00010000:2.0:1734016145.112340:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d1cd lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.112348:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d1cd lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.112416:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf180/0x37cab3886ce5d1d4 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.112422:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf180/0x37cab3886ce5d1d4 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.115859:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334637824 00000100:00100000:2.0:1734016145.115873:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334637824 00000100:00100000:2.0:1734016145.115938:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334637952 00000100:00100000:2.0:1734016145.115940:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334637952 00000100:00100000:2.0:1734016145.115962:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14711 00000100:00100000:2.0:1734016145.115974:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334637824:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016145.116001:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012b714000/0x37cab3886ce5d1b8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31785c0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.116016:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012b714000/0x37cab3886ce5d1b8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31785c0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.116055:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016145.116058:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 5166370 mtime: 1734016145 atime: 1734016144 ctime: 1734016145 blocks: 224 00000020:00010000:2.0:1734016145.116062:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016145.116064:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 5166370 mtime: 1734016145 atime: 1734016144 ctime: 1734016145 blocks: 224 00000100:00100000:2.0:1734016145.116145:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334637824:12345-192.168.201.152@tcp:106: Request processed in 171us (751us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.116153:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14711 00000100:00100000:2.0:1734016145.116179:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14712 00000100:00100000:2.0:1734016145.116182:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334637952:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016145.116189:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.116197:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.116207:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016145.116208:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016144 ctime: 1734016145 blocks: 128 00000020:00010000:2.0:1734016145.116212:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016145.116213:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016144 ctime: 1734016145 blocks: 128 00000100:00100000:2.0:1734016145.116287:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334637952:12345-192.168.201.152@tcp:106: Request processed in 103us (733us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.116293:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14712 00010000:00010000:2.0:1734016145.120007:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf180/0x37cab3886ce5d1d4 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.120031:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf180/0x37cab3886ce5d1d4 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.120045:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf180/0x37cab3886ce5d1d4 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.120053:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf180/0x37cab3886ce5d1d4 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.120058:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf180/0x37cab3886ce5d1d4 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.120075:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016145.120083:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf180/0x37cab3886ce5d1d4 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.120088:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf180/0x37cab3886ce5d1d4 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.120094:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf180/0x37cab3886ce5d1d4 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.120100:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbf180 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016145.120107:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf180/0x37cab3886ce5d1d4 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.120117:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016145.120123:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf180/0x37cab3886ce5d1d4 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.120128:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf180/0x37cab3886ce5d1d4 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.120830:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332528768:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016145.120854:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d1cd lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.120861:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d1cd lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.120868:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d1cd lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.120873:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d1cd lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.120878:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d1cd lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.120887:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5166370, setting rss=5166370; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016145.121031:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d1cd lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.121039:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d1cd lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.121044:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d1cd lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.121048:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbd8c0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016145.121051:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d1cd lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.121057:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016145.121060:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d1cd lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.121066:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d1cd lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.121084:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332528640:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016145.121794:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016145.121809:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800b341ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332528896:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016145.125799:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334638080 00000100:00100000:2.0:1734016145.125816:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334638080 00000100:00100000:2.0:1734016145.125874:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334638208 00000100:00100000:2.0:1734016145.125876:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334638208 00000100:00100000:2.0:1734016145.125892:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14713 00000100:00100000:2.0:1734016145.125899:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334638080:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016145.125920:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012b714000/0x37cab3886ce5d1b8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31785c0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.125934:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012b714000/0x37cab3886ce5d1b8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31785c0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.125964:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016145.125966:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 5166370 mtime: 1734016145 atime: 1734016144 ctime: 1734016145 blocks: 224 00000020:00010000:2.0:1734016145.125970:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016145.125971:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 5166370 mtime: 1734016145 atime: 1734016144 ctime: 1734016145 blocks: 224 00000100:00100000:2.0:1734016145.126036:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334638080:12345-192.168.201.152@tcp:106: Request processed in 137us (685us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.126042:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14713 00000100:00100000:2.0:1734016145.126077:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14714 00000100:00100000:2.0:1734016145.126081:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334638208:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016145.126087:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.126094:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.126104:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016145.126104:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016144 ctime: 1734016145 blocks: 128 00000020:00010000:2.0:1734016145.126108:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016145.126110:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016144 ctime: 1734016145 blocks: 128 00000100:00100000:2.0:1734016145.126125:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334638208:12345-192.168.201.152@tcp:106: Request processed in 43us (480us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.126131:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14714 00010000:00010000:2.0:1734016145.130340:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964f180/0x37cab3886ce5d1db lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.130351:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964f180/0x37cab3886ce5d1db lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.130358:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964f180/0x37cab3886ce5d1db lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.130364:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964f180/0x37cab3886ce5d1db lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.130371:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964f180/0x37cab3886ce5d1db lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.130378:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5166370, setting rss=5166370; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016145.130387:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964f180/0x37cab3886ce5d1db lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.130392:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964f180/0x37cab3886ce5d1db lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.130397:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964f180/0x37cab3886ce5d1db lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.130402:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a964f180 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016145.130406:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964f180/0x37cab3886ce5d1db lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.130411:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016145.130414:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964f180/0x37cab3886ce5d1db lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.130418:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964f180/0x37cab3886ce5d1db lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.130441:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332528896:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016145.156826:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334638336 00000100:00100000:2.0:1734016145.156841:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334638336 00000100:00100000:2.0:1734016145.156913:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14715 00000100:00100000:2.0:1734016145.156925:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334638336:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016145.156949:0:4729:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012b714000/0x37cab3886ce5d1b8 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d31785c0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.157062:0:4729:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016145.157071:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334638336:12345-192.168.201.152@tcp:104: Request processed in 147us (563us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.157081:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14715 00000100:00100000:2.0:1734016145.165134:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000020:00010000:2.0:1734016145.262903:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8218 (ffff8800a72d9878/0) at cl_lock_release():233 00000020:00010000:2.0:1734016145.262927:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8218 (ffff8800a72d9878/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.262932:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b352ed30 (ffff8800a72d92d8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016145.262951:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b714d80/0x37cab3886ce5d1f0 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3338240->3350527) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178606 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.262971:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b714d80/0x37cab3886ce5d1f0 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3338240->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178606 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.263000:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8218 (ffff8800a72d9878/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.263005:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b352ed30 (ffff8800a72d9878/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016145.263024:0:10366:0:(vvp_io.c:533:vvp_io_advance()) advancing 6976 bytes 00000080:00200000:2.0:1734016145.263049:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016145.263055:0:10366:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 1 io complete with rc: 0, result: 6976, restart: 0 00000080:00200000:2.0:1734016145.263063:0:10366:0:(file.c:2044:ll_file_io_generic()) iot: 1, result: 6976 00000080:00200008:2.0:1734016145.263116:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016145.263127:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016145.263162:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016145.263193:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d1c6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31785d5 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016145.263208:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d1c6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31785d5 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016145.263300:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016145.263307:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d1c6 00000080:00010000:2.0:1734016145.263314:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d1c6 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016145.263318:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d1c6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31785d5 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016145.263329:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d1c6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31785d5 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016145.263361:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016145.263373:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016145.263394:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016145.263397:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2930, osc lock ffff8800aafcf300, flags 0x400001002 00000008:00010000:2.0:1734016145.263405:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016145.263407:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2970, osc lock ffff8800aafcf0a0, flags 0x400001002 00010000:00010000:2.0:1734016145.263424:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b714d80/0x37cab3886ce5d1f0 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3338240->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178606 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.263434:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b714d80/0x37cab3886ce5d1f0 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3338240->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178606 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.263450:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b714d80/0x37cab3886ce5d1f0 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3338240->3350527) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178606 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.263457:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b714d80/0x37cab3886ce5d1f0 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3338240->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178606 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.263488:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d1f7 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.263497:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d1f7 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.263687:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016145.263726:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332529920:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016145.271803:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334638592 00000100:00100000:2.0:1734016145.271819:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334638592 00000100:00100000:2.0:1734016145.271922:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14717 00000100:00100000:2.0:1734016145.271932:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334638592:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016145.271965:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.271981:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.272043:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016145.272048:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016144 ctime: 1734016145 blocks: 144 00000020:00010000:2.0:1734016145.272052:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016145.272053:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016144 ctime: 1734016145 blocks: 144 00000100:00100000:2.0:1734016145.272162:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334638592:12345-192.168.201.152@tcp:106: Request processed in 231us (634us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.272170:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14717 00010000:00010000:2.0:1734016145.275132:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d1f7 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.275167:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d1f7 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.275185:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d1f7 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.275199:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d1f7 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.275205:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d1f7 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.275278:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016145.276318:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d1f7 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.276330:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d1f7 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.276340:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d1f7 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.276347:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbed00 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016145.276353:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d1f7 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.276370:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016145.276379:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d1f7 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.276387:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d1f7 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.287599:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332529920:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016145.292504:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334638720 00000100:00100000:2.0:1734016145.292520:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334638720 00000100:00100000:2.0:1734016145.292601:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14718 00000100:00100000:2.0:1734016145.292612:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334638720:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016145.292636:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.292649:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.292696:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016145.292698:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016144 ctime: 1734016145 blocks: 144 00000020:00010000:2.0:1734016145.292702:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016145.292703:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016144 ctime: 1734016145 blocks: 144 00000100:00100000:2.0:1734016145.292789:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334638720:12345-192.168.201.152@tcp:106: Request processed in 178us (1085us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.292798:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14718 00000080:00200000:2.0:1734016145.298267:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9360674 i_blocks 368 00000020:00010000:2.0:1734016145.298277:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70ddff0 (ffff8800a72d9878/0) at cl_lock_release():233 00000020:00010000:2.0:1734016145.298281:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70ddff0 (ffff8800a72d9878/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.298286:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676330 (ffff8800a72d92d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.298291:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676370 (ffff88012db7cea8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.298293:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70ddff0 (ffff8800a72d9878/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.298295:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676330 (ffff8800a72d9878/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.298303:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676370 (ffff8800a72d9878/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016145.298318:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016145.298330:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9360674 00000080:00200008:2.0:1734016145.298352:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016145.298359:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016145.298384:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016145.298400:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff8800a964fcc0/0x37cab3886ce5d1e9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31785ff expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016145.298413:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff8800a964fcc0/0x37cab3886ce5d1e9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31785ff expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016145.298431:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016145.298434:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d1e9 00000080:00010000:2.0:1734016145.298441:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d1e9 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016145.298443:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff8800a964fcc0/0x37cab3886ce5d1e9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31785ff expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016145.298450:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff8800a964fcc0/0x37cab3886ce5d1e9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31785ff expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016145.298470:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016145.298478:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016145.298513:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016145.298516:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e29f0, osc lock ffff8800aafcfb50, flags 0x400001002 00000008:00010000:2.0:1734016145.298522:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016145.298523:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2a30, osc lock ffff8800aafcf8f0, flags 0x400001002 00010000:00010000:2.0:1734016145.298549:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbed00/0x37cab3886ce5d205 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.298560:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbed00/0x37cab3886ce5d205 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.299922:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.299936:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.299949:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.299959:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.313363:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334638848 00000100:00100000:2.0:1734016145.313382:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334638848 00000100:00100000:2.0:1734016145.313456:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14719 00000100:00100000:2.0:1734016145.313468:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334638848:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016145.313495:0:4728:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012b717840/0x37cab3886ce5d1bf lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d31785c7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.313617:0:4728:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016145.313625:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334638848:12345-192.168.201.152@tcp:104: Request processed in 158us (432us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.313634:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14719 00000100:00100000:2.0:1734016145.324018:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334638976 00000100:00100000:2.0:1734016145.324034:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334638976 00000100:00100000:2.0:1734016145.324112:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14720 00000100:00100000:2.0:1734016145.324122:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7025:x1818247334638976:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016145.324139:0:4729:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20240/0x37cab3886ce5d20c lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [2498560->2510847] (req 2498560->2510847) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178622 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.324152:0:4729:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20240/0x37cab3886ce5d20c lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [2498560->2510847] (req 2498560->2510847) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178622 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.324160:0:4729:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20240/0x37cab3886ce5d20c lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2498560->2510847) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d3178622 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.324172:0:4729:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20240/0x37cab3886ce5d20c lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2498560->2510847) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d3178622 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.324182:0:4729:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20240/0x37cab3886ce5d20c lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2498560->2510847) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d3178622 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.324193:0:4729:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20240/0x37cab3886ce5d20c lrc: 6/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2498560->2510847) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d3178622 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.324236:0:4729:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a6c20240) 00000100:00100000:2.0:1734016145.324306:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7025:x1818247334638976:12345-192.168.201.152@tcp:105: Request processed in 184us (417us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.324313:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14720 00010000:00010000:2.0:1734016145.433860:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b714d80/0x37cab3886ce5d1f0 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3338240->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178606 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.433877:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4194304 4198399) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b714d80/0x37cab3886ce5d1f0 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3338240->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178606 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.435148:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b714d80/0x37cab3886ce5d1f0 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3338240->3350527) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178606 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.435161:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b714d80/0x37cab3886ce5d1f0 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3338240->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178606 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.435484:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8218 (ffff8800a72d9878/0) at cl_lock_release():233 00000020:00010000:2.0:1734016145.435488:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8218 (ffff8800a72d9878/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.435493:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f87b0 (ffff88012db7cea8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016145.435501:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20240/0x37cab3886ce5d20c lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2498560->2510847) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178622 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.435509:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20240/0x37cab3886ce5d20c lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2498560->2510847) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178622 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.435521:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8218 (ffff8800a72d9878/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.435523:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f87b0 (ffff8800a72d9878/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016145.435545:0:10366:0:(vvp_io.c:533:vvp_io_advance()) advancing 7161 bytes 00000080:00200000:2.0:1734016145.435562:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016145.435566:0:10366:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 1 io complete with rc: 0, result: 7161, restart: 0 00000080:00200000:2.0:1734016145.435573:0:10366:0:(file.c:2044:ll_file_io_generic()) iot: 1, result: 7161 00000080:00200008:2.0:1734016145.435619:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016145.435628:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016145.435654:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016145.435671:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff8800a964fcc0/0x37cab3886ce5d1e9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31785ff expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016145.435680:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff8800a964fcc0/0x37cab3886ce5d1e9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31785ff expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016145.435695:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016145.435698:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d1e9 00000080:00010000:2.0:1734016145.435704:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d1e9 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016145.435706:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff8800a964fcc0/0x37cab3886ce5d1e9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31785ff expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016145.435712:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff8800a964fcc0/0x37cab3886ce5d1e9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31785ff expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016145.435744:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016145.435760:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016145.435783:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016145.435788:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c30, osc lock ffff8800aafcfa20, flags 0x400001002 00000008:00010000:2.0:1734016145.435795:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016145.435797:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c70, osc lock ffff8800aafcf690, flags 0x400001002 00010000:00010000:2.0:1734016145.435813:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbed00/0x37cab3886ce5d205 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d317861b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.435821:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbed00/0x37cab3886ce5d205 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d317861b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.435833:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbed00/0x37cab3886ce5d205 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d317861b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.435840:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbed00/0x37cab3886ce5d205 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d317861b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.435877:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d213 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.435885:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d213 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.435988:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332530944:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016145.436126:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:2.0:1734016145.436141:0:1970:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8800aa7e2240 00010000:00010000:2.0:1734016145.436185:0:1970:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4194304 4198399) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b714d80/0x37cab3886ce5d1f0 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3338240->3350527) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178606 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016145.436266:0:1970:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73cbb80 - object 0x280000400:7 offset 4194304<>5169152 00000100:00100000:2.0:1734016145.436284:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016145.436290:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332531072:192.168.201.152@tcp:3:fsx.0 00010000:00010000:2.0:1734016145.448553:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d213 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.448584:0:1970:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d213 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xb46129d8d3178629 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.448604:0:1970:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d213 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d3178629 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.448620:0:1970:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d213 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xb46129d8d3178629 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.448637:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d213 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178629 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.448664:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d213 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d3178629 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.448674:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d213 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d3178629 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.448680:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d213 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d3178629 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.448688:0:1970:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd8c0/0x37cab3886ce5d213 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d3178629 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.448716:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332530944:192.168.201.152@tcp:101:fsx.0 00000008:00010000:2.0:1734016145.457284:0:1970:0:(osc_cache.c:824:osc_extent_finish()) ### no dlm lock: extent: ffff8800aa7e2240 00000008:00010000:2.0:1734016145.459893:0:1970:0:(osc_cache.c:484:__osc_extent_remove()) ### no dlm lock: extent: ffff8800aa7e2240 00000100:00100000:2.0:1734016145.459914:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332531072:192.168.201.152@tcp:3:fsx.0 00000100:00100000:2.0:1734016145.474860:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016145.476694:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016145.476733:0:9532:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016145.476747:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334639360 00000100:00100000:2.0:1734016145.476750:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334639360 00000100:00100000:2.0:1734016145.476768:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14723 00000100:00100000:2.0:1734016145.476773:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012b52fc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334639360:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016145.476783:0:9532:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b714d80/0x37cab3886ce5d1f0 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3338240->3350527) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d3178606 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.478138:0:9532:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:2.0:1734016145.478148:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012b52fc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334639360:12345-192.168.201.152@tcp:104: Request processed in 1371us (1452us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.478155:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14723 00000100:00100000:2.0:1734016145.486890:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016145.491122:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016145.491153:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016145.493326:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14726 00000100:00100000:2.0:1734016145.493334:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73c9f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334639744:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016145.493345:0:4729:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20240/0x37cab3886ce5d20c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2498560->2510847) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d3178622 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.493387:0:4729:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016145.493392:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73c9f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334639744:12345-192.168.201.152@tcp:104: Request processed in 59us (2238us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.493400:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14726 00000100:00100000:2.0:1734016145.503594:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016145.522147:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 1477430 i_blocks 96 00000020:00010000:2.0:1734016145.522155:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8b78 (ffff8800a72d95a8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016145.522161:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8b78 (ffff8800a72d95a8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.522168:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40a030 (ffff8800a72d9e18/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016145.522181:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b716ac0/0x37cab3886ce5d21a lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178637 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.522195:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b716ac0/0x37cab3886ce5d21a lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178637 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.522256:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40a070 (ffff8800a72d9ab8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016145.522263:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20fc0/0x37cab3886ce5d221 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317863e expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.522271:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20fc0/0x37cab3886ce5d221 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317863e expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.522281:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8b78 (ffff8800a72d95a8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.522284:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40a030 (ffff8800a72d95a8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.522290:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40a070 (ffff8800a72d95a8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016145.522316:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016145.522338:0:10366:0:(llite_lib.c:2534:ll_setattr_raw()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8) current size 1477430, valid attrs 2168, mode 8800, uid -1451891104, gid -30720, new size 1477430, atime 1734016145.000000000, mtime 1734016145.450000000, ctime 1734016145.450000000, rc 0 00000080:00200008:2.0:1734016145.522371:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016145.522381:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016145.522409:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016145.522434:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b62b8800 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:2.0:1734016145.522497:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d228 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016145.522505:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d228 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016145.522517:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d228 lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00000100:00100000:2.0:1734016145.522550:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332532608:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016145.522623:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2600 going to sleep for 11 seconds 00000100:00100000:2.0:1734016145.528014:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332532608:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016145.528073:0:10366:0:(ldlm_request.c:706:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d228 lrc: 4/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d317864c expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016145.528090:0:10366:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d228 lrc: 4/1,0 mode: --/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d317864c expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016145.528109:0:10366:0:(ldlm_lock.c:1060:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d228 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d317864c expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016145.528122:0:10366:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d228 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d317864c expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016145.528139:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d228 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d317864c expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016145.528149:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d228 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d317864c expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1734016145.528156:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d228 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d317864c expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1734016145.528175:0:10366:0:(mdc_locks.c:822:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff8800a73cb100 x1818247332532608/t0(0) o101->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 576/664 e 0 to 0 dl 1734016161 ref 1 fl Complete:RQU/200/0 rc 0/0 job:'fsx.0' uid:0 gid:0 00000002:00010000:2.0:1734016145.528189:0:10366:0:(mdc_locks.c:926:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 80 ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d228 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d317864c expref: -99 pid: 10366 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1734016145.528205:0:10366:0:(mdc_locks.c:1262:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d228 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb46129d8d317864c expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016145.528278:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b62b8800 type 13 mode 4 res 8589935617/12 (0 0) 00000080:00200000:2.0:1734016145.528390:0:10366:0:(file.c:6447:ll_layout_conf()) [0x200000401:0xc:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1734016145.528398:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d228 00000080:00010000:2.0:1734016145.528407:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016145.528411:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d228 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016145.528414:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d228 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d317864c expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016145.528420:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d228 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d317864c expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016145.528472:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016145.528492:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016145.528518:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016145.528521:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2630, osc lock ffff8800aafced10, flags 0x400001002 00000008:00010000:2.0:1734016145.528541:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016145.528543:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2670, osc lock ffff8800aafcee40, flags 0x400001002 00010000:00010000:2.0:1734016145.528578:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d22f lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.528590:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d22f lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.528665:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d236 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.528672:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d236 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.538193:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d236 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.538271:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d236 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.538291:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d236 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.538299:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d236 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.538308:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d236 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.538332:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016145.538346:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d236 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.538356:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d236 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.538366:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d236 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.538374:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbd680 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016145.538385:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d236 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.538403:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016145.538414:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d236 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.538420:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d236 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.538494:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332532864:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016145.538518:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d22f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.538526:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d22f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.538543:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d22f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.538549:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d22f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.538554:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d22f lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.538561:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 1477430, setting rss=1477430; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016145.538578:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d22f lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.538583:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d22f lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.538588:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d22f lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.538592:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbcd80 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016145.538594:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d22f lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.538599:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016145.538602:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d22f lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.538606:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d22f lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.538629:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332532736:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016145.550425:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964f840/0x37cab3886ce5d244 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.550467:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964f840/0x37cab3886ce5d244 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.550488:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964f840/0x37cab3886ce5d244 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.550507:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964f840/0x37cab3886ce5d244 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.550515:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964f840/0x37cab3886ce5d244 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.550539:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016145.550551:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964f840/0x37cab3886ce5d244 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.550559:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964f840/0x37cab3886ce5d244 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.550567:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964f840/0x37cab3886ce5d244 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.550580:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a964f840 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016145.550588:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964f840/0x37cab3886ce5d244 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.550605:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016145.550614:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964f840/0x37cab3886ce5d244 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.550621:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a964f840/0x37cab3886ce5d244 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.550689:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332533120:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016145.550715:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964fcc0/0x37cab3886ce5d23d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.550726:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964fcc0/0x37cab3886ce5d23d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.550734:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964fcc0/0x37cab3886ce5d23d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.550741:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964fcc0/0x37cab3886ce5d23d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.550747:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964fcc0/0x37cab3886ce5d23d lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.550759:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 1477430, setting rss=1477430; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016145.550769:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964fcc0/0x37cab3886ce5d23d lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.550774:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964fcc0/0x37cab3886ce5d23d lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.550781:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964fcc0/0x37cab3886ce5d23d lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.550786:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a964fcc0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016145.550789:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964fcc0/0x37cab3886ce5d23d lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.550796:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016145.550799:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964fcc0/0x37cab3886ce5d23d lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.550805:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964fcc0/0x37cab3886ce5d23d lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.550818:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332532992:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016145.562183:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334640512 00000100:00100000:2.0:1734016145.562225:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334640512 00000100:00100000:2.0:1734016145.562318:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14732 00000100:00100000:2.0:1734016145.562330:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334640512:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016145.562362:0:4728:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b716ac0/0x37cab3886ce5d21a lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d3178637 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.562512:0:4728:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016145.562520:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334640512:12345-192.168.201.152@tcp:104: Request processed in 191us (2108us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.562535:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14732 00000100:00100000:2.0:1734016145.575546:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334640640 00000100:00100000:2.0:1734016145.575560:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334640640 00000100:00100000:2.0:1734016145.575609:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14733 00000100:00100000:2.0:1734016145.575619:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7025:x1818247334640640:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016145.575636:0:9532:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [4677632->4685823] (req 4677632->4685823) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.575648:0:9532:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [4677632->4685823] (req 4677632->4685823) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.575657:0:9532:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.575666:0:9532:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.575676:0:9532:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.575695:0:9532:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.575743:0:9532:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a964ed00) 00000100:00100000:2.0:1734016145.575810:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7025:x1818247334640640:12345-192.168.201.152@tcp:105: Request processed in 192us (430us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.575818:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14733 00000100:00100000:2.0:1734016145.581724:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334640768 00000100:00100000:2.0:1734016145.581743:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334640768 00000100:00100000:2.0:1734016145.581811:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14734 00000100:00100000:2.0:1734016145.581824:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ca80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334640768:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016145.581850:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20fc0/0x37cab3886ce5d221 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317863e expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.581863:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20fc0/0x37cab3886ce5d221 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317863e expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.581899:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016145.581902:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016145 atime: 1734016144 ctime: 1734016145 blocks: 0 00000020:00010000:2.0:1734016145.581905:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016145.581906:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016145 atime: 1734016144 ctime: 1734016145 blocks: 0 00000100:00100000:2.0:1734016145.581976:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ca80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334640768:12345-192.168.201.152@tcp:106: Request processed in 155us (578us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.581982:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14734 00000080:00200000:2.0:1734016145.586411:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8876504 i_blocks 96 00000020:00010000:2.0:1734016145.586420:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd6d0 (ffff8800a72d95a8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016145.586424:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd6d0 (ffff8800a72d95a8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.586432:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40ab70 (ffff8800a72d9ab8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.586436:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40abb0 (ffff88012db7cea8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.586439:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd6d0 (ffff8800a72d95a8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.586441:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40ab70 (ffff8800a72d95a8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.586447:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40abb0 (ffff8800a72d95a8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016145.586475:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016145.586713:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016145.586723:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016145.586741:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016145.586774:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016145.586777:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2cf0, osc lock ffff8800aafcfdb0, flags 0x400001002 00000008:00010000:2.0:1734016145.586785:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016145.586786:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2d30, osc lock ffff8800aafce980, flags 0x400001002 00010000:00010000:2.0:1734016145.586810:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.586827:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.586838:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.586848:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.586895:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d259 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.586902:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d259 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.587743:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016145.587771:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332533632:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016145.590382:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334640896 00000100:00100000:2.0:1734016145.590395:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334640896 00000100:00100000:2.0:1734016145.590472:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14735 00000100:00100000:2.0:1734016145.590483:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334640896:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016145.590506:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20fc0/0x37cab3886ce5d221 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317863e expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.590519:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20fc0/0x37cab3886ce5d221 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317863e expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.590564:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016145.590567:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016145 atime: 1734016144 ctime: 1734016145 blocks: 0 00000020:00010000:2.0:1734016145.590570:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016145.590571:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016145 atime: 1734016144 ctime: 1734016145 blocks: 0 00000100:00100000:2.0:1734016145.590642:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334640896:12345-192.168.201.152@tcp:106: Request processed in 160us (397us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.590648:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14735 00010000:00010000:2.0:1734016145.593651:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d259 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.593679:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d259 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.593694:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d259 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.593700:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d259 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.593707:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d259 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.593728:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 0, setting rss=0; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016145.594301:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d259 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.594308:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d259 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.594316:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d259 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.594325:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbdf80 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016145.594330:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d259 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.594344:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016145.594349:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d259 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.594360:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d259 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.594673:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332533632:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016145.600795:0:4728:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016145.600817:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334641024 00000100:00100000:2.0:1734016145.600829:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334641024 00000100:00100000:2.0:1734016145.600883:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14736 00000100:00100000:2.0:1734016145.600891:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012b52f450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7694:x1818247334641024:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016145.600909:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20fc0/0x37cab3886ce5d221 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317863e expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.600920:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20fc0/0x37cab3886ce5d221 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317863e expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.600950:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016145.600953:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016145 atime: 1734016144 ctime: 1734016145 blocks: 0 00000020:00010000:2.0:1734016145.600956:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016145.600957:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 0 mtime: 1734016145 atime: 1734016144 ctime: 1734016145 blocks: 0 00000100:00100000:2.0:1734016145.601017:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012b52f450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7694:x1818247334641024:12345-192.168.201.152@tcp:106: Request processed in 126us (359us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.601025:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14736 00000080:00200000:2.0:1734016145.604568:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8876504 i_blocks 96 00000020:00010000:2.0:1734016145.604578:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd6d0 (ffff8800a72d95a8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016145.604582:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd6d0 (ffff8800a72d95a8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.604590:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86763f0 (ffff8800a72d9ab8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.604595:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676430 (ffff88012db7cea8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.604598:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd6d0 (ffff8800a72d95a8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.604600:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86763f0 (ffff8800a72d95a8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.604607:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676430 (ffff8800a72d95a8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016145.604628:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016145.604754:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 1234 00000080:00200000:2.0:1734016145.604767:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016145.604792:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016145.604813:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1234, 1234] 00000008:00010000:2.0:1734016145.604837:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016145.604841:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8a30, osc lock ffff8800aafcfc80, flags 0x0 00010000:00010000:2.0:1734016145.604878:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.604886:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [860160->864255] (req 860160->864255) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.604925:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [860160->864255] (req 860160->864255) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.604956:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332533888:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016145.605137:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2240 going to sleep for 11 seconds 00000100:00100000:2.0:1734016145.608835:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334641152 00000100:00100000:2.0:1734016145.608853:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334641152 00000100:00100000:2.0:1734016145.608947:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14737 00000100:00100000:2.0:1734016145.608957:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7694:x1818247334641152:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016145.608976:0:9532:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c20fc0/0x37cab3886ce5d221 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d317863e expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.609760:0:9532:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016145.609771:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7694:x1818247334641152:12345-192.168.201.152@tcp:104: Request processed in 812us (1044us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.609779:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14737 00000100:00100000:2.0:1734016145.615061:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332533888:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016145.615114:0:10366:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [860160->864255] (req 860160->864255) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.615128:0:10366:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [860160->864255] (req 860160->864255) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.615137:0:10366:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [860160->864255] (req 860160->864255) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.615275:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334641280 00000100:00100000:2.0:1734016145.615287:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334641280 00000100:00100000:2.0:1734016145.615361:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14738 00000100:00100000:2.0:1734016145.615368:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7025:x1818247334641280:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016145.615378:0:9532:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [860160->864255] (req 860160->864255) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.615387:0:9532:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [860160->864255] (req 860160->864255) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.615394:0:9532:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.615402:0:9532:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.615410:0:9532:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.615418:0:9532:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.615446:0:9532:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff88012dbbdd40) 00000100:00100000:2.0:1734016145.616949:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7025:x1818247334641280:12345-192.168.201.152@tcp:105: Request processed in 1570us (2024us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.616962:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14738 00000020:00010000:2.0:1734016145.632797:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8b78 (ffff8800a72d95a8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016145.632801:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8b78 (ffff8800a72d95a8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.632805:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8a30 (ffff88012db7cea8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016145.632814:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.632825:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.632846:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8b78 (ffff8800a72d95a8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.632848:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8a30 (ffff8800a72d95a8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016145.632875:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016145.632903:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016145.632920:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016145.632937:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1234, 1234] 00000008:00010000:2.0:1734016145.632957:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016145.632963:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8a30, osc lock ffff8800aafcfc80, flags 0x0 00010000:00010000:2.0:1734016145.632985:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.632999:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (860160 864255) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.633010:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.633016:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.633022:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.633068:0:10366:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800aa7e2240 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016145.633086:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 1, dirtied 1 00000020:00010000:2.0:1734016145.633098:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8b78 (ffff8800a72d95a8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016145.633101:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8b78 (ffff8800a72d95a8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.633102:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8a30 (ffff88012db7cea8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016145.633106:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.633112:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.633119:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8b78 (ffff8800a72d95a8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.633121:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8a30 (ffff8800a72d95a8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016145.633127:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016145.633203:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 1235 00000080:00200000:2.0:1734016145.633210:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016145.633214:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016145.633251:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1235, 1235] 00000008:00010000:2.0:1734016145.633257:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016145.633258:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8a30, osc lock ffff8800aafcfc80, flags 0x0 00010000:00010000:2.0:1734016145.633264:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.633269:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (864256 868351) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.633276:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.633281:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.633285:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.633336:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.633345:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4194304 4198399) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.633353:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 6/1,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.633358:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 6/1,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.633362:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.634076:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.634087:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.638539:0:1971:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e2b40 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.638591:0:1971:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2b40 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.638636:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332534656:192.168.201.152@tcp:3:fsx.0 00000080:00200000:2.0:1734016145.649780:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1734016145.649871:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332535040:192.168.201.152@tcp:44:fsx.0 00000100:00100000:2.0:1734016145.650064:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2b40 going to sleep for 11 seconds 00000100:00100000:2.0:1734016145.653545:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332535040:192.168.201.152@tcp:44:fsx.0 00000080:00200000:2.0:1734016145.653593:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016145.653612:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1734016145.653691:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016145.653711:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332535168:192.168.201.152@tcp:16:fsx.0 00000100:00100000:2.0:1734016145.657659:0:1970:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8800b3cc4800: committing for last_committed 4294968276 gen 2 00000100:00100000:2.0:1734016145.658137:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332535168:192.168.201.152@tcp:16:fsx.0 00000020:00010000:2.0:1734016145.666661:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8e48 (ffff8800a72d9bd8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016145.666667:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8e48 (ffff8800a72d9bd8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.666672:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b352e8b0 (ffff8800a72d92d8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016145.666686:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.666697:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.666715:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8e48 (ffff8800a72d9bd8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.666718:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b352e8b0 (ffff8800a72d9bd8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016145.666747:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016145.666775:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016145.666797:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016145.666817:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [8, 8] 00000008:00010000:2.0:1734016145.666835:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016145.666841:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef81b0, osc lock ffff8800aafcf560, flags 0x0 00010000:00010000:2.0:1734016145.666870:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.666884:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (32768 36863) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.666897:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.666904:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.666909:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.666951:0:10366:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800aa7e2b40 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016145.666964:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 1, dirtied 1 00000020:00010000:2.0:1734016145.666972:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8e48 (ffff8800a72d9bd8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016145.666973:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8e48 (ffff8800a72d9bd8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.666974:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef81b0 (ffff8800a72d92d8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016145.666976:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.666981:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.666986:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8e48 (ffff8800a72d9bd8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.666987:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef81b0 (ffff8800a72d9bd8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016145.666994:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016145.667025:0:10366:0:(file.c:5185:ll_fsync()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8), start 32768, end 36863, datasync 1 00000080:00200000:2.0:1734016145.667030:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016145.667033:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1734016145.667041:0:10366:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e2b40 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.667045:0:10366:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa40a600 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.667052:0:10366:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800aa7e2b40 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.667156:0:10366:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800aa40a600 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.667186:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (32768 36863) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016145.667293:0:10366:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73c8000 - object 0x280000400:7 offset 32768<>4682200 00000080:00200000:2.0:1734016145.677755:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016145.677783:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016145.677796:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1734016145.684428:0:1971:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8800b3cc4800: committing for last_committed 4294969910 gen 1 00000100:00100000:2.0:1734016145.684595:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332535680:192.168.201.152@tcp:16:fsx.0 00000080:00200000:2.0:1734016145.701931:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8876504 i_blocks 136 00000020:00010000:2.0:1734016145.701939:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016145.701943:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.701949:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40a6f0 (ffff8800a72d9878/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.701952:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40a730 (ffff8800a72d9098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.701955:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.701957:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40a6f0 (ffff8800a72d9bd8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.701961:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40a730 (ffff8800a72d9bd8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016145.701972:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016145.701994:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016145.702000:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016145.702009:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016145.702021:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016145.702024:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2b70, osc lock ffff8800aafcf560, flags 0x400001002 00000008:00010000:2.0:1734016145.702029:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016145.702030:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2bb0, osc lock ffff8800aafce720, flags 0x400001002 00010000:00010000:2.0:1734016145.702049:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5d283 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.702056:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5d283 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.702103:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016145.702114:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332536192:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016145.702170:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d28a lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.702177:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d28a lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.702201:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332536320:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016145.713875:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5d283 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.713904:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5d283 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.713918:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5d283 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.713925:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5d283 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.713931:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5d283 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.713953:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4682200, setting rss=4682200; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016145.713965:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5d283 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.713971:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5d283 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.713980:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5d283 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.713988:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbcfc0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016145.713994:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5d283 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.714006:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016145.714013:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5d283 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.714019:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcfc0/0x37cab3886ce5d283 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.714482:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332536192:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016145.716154:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d28a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.716174:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d28a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.716188:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d28a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.716194:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d28a lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.716201:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d28a lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.716260:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 868352, setting rss=868352; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016145.716277:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d28a lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.716283:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d28a lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.716290:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d28a lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.716298:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbc000 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016145.716302:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d28a lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.716316:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016145.716322:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d28a lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.716329:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d28a lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.716363:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332536320:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016145.720506:0:9532:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016145.720550:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334641920 00000100:00100000:2.0:1734016145.720568:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334641920 00000100:00100000:2.0:1734016145.721208:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14743 00000100:00100000:2.0:1734016145.721504:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7039050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334641920:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016145.721539:0:9532:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d267 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 860160->864255) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d317868b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.722167:0:9532:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016145.722178:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7039050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334641920:12345-192.168.201.152@tcp:104: Request processed in 678us (2510us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.722189:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14743 00000100:00100000:2.0:1734016145.729730:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334642048 00000100:00100000:2.0:1734016145.729749:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334642048 00000100:00100000:2.0:1734016145.729829:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14744 00000100:00100000:2.0:1734016145.729839:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7025:x1818247334642048:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016145.729865:0:9532:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d291 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [2953216->2961407] (req 2953216->2961407) gid 0 flags: 0x0 nid: local remote: 0xb46129d8d31786bc expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.729882:0:9532:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d291 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [2953216->2961407] (req 2953216->2961407) gid 0 flags: 0x0 nid: local remote: 0xb46129d8d31786bc expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.729894:0:9532:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d291 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d31786bc expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.729906:0:9532:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d291 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x10080000000000 nid: local remote: 0xb46129d8d31786bc expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.729922:0:9532:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d291 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x10080000000000 nid: local remote: 0xb46129d8d31786bc expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.729935:0:9532:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d291 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x80000000000 nid: local remote: 0xb46129d8d31786bc expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.729961:0:9532:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a6c218c0) 00000100:00100000:2.0:1734016145.730037:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7025:x1818247334642048:12345-192.168.201.152@tcp:105: Request processed in 199us (422us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.730043:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14744 00000100:00100000:2.0:1734016145.734116:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334642176 00000100:00100000:2.0:1734016145.734126:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334642176 00000100:00100000:2.0:1734016145.734182:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14745 00000100:00100000:2.0:1734016145.734189:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334642176:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016145.734208:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.734258:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.734315:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 122, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016145.734318:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4682200 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 120 00000020:00010000:2.0:1734016145.734321:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016145.734322:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4682200 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 120 00000100:00100000:2.0:1734016145.734384:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334642176:12345-192.168.201.152@tcp:106: Request processed in 194us (565us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.734393:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14745 00000080:00200000:2.0:1734016145.741722:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8876504 i_blocks 136 00000020:00010000:2.0:1734016145.741732:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016145.741737:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.741744:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86764b0 (ffff8800a72d9098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.741750:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86764f0 (ffff8800a72d9878/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.741755:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.741757:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86764b0 (ffff8800a72d9bd8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.741763:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86764f0 (ffff8800a72d9bd8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016145.741776:0:10366:0:(vvp_io.c:899:vvp_io_read_start()) count:5065 iocb pos:7149534 00000100:00100000:2.0:1734016145.741907:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00010000:2.0:1734016145.741923:0:1971:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e2f00 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d291 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786bc expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.741963:0:1971:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2953216 2957311) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d291 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786bc expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016145.742012:0:1971:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73c8000 - object 0x240000400:7 offset 2953216<>2957312 00000100:00100000:2.0:1734016145.742039:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332536960:192.168.201.152@tcp:3:fsx.0 00000008:00010000:2.0:1734016145.748485:0:1971:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e2f00 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d291 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786bc expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.748658:0:1971:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2f00 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d291 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786bc expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.748699:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332536960:192.168.201.152@tcp:3:fsx.0 00000100:00100000:2.0:1734016145.760247:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334642304 00000100:00100000:2.0:1734016145.760267:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334642304 00000100:00100000:2.0:1734016145.760845:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14746 00000100:00100000:2.0:1734016145.760859:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334642304:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016145.760887:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.760904:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.760947:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 122, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016145.760949:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4682200 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 120 00000020:00010000:2.0:1734016145.760953:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016145.760954:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4682200 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 120 00000100:00100000:2.0:1734016145.761025:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334642304:12345-192.168.201.152@tcp:106: Request processed in 170us (1251us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.761034:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14746 00000080:00200000:2.0:1734016145.764622:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8876504 i_blocks 136 00000020:00010000:2.0:1734016145.764630:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016145.764633:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.764639:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86760f0 (ffff8800a72d9098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.764645:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676130 (ffff8800a72d9878/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.764648:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.764650:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86760f0 (ffff8800a72d9bd8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.764658:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676130 (ffff8800a72d9bd8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016145.764677:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016145.764712:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016145.764719:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016145.764733:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016145.764757:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016145.764759:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f30, osc lock ffff8800aafcf7c0, flags 0x400001002 00000008:00010000:2.0:1734016145.764765:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016145.764766:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f70, osc lock ffff8800aafce5f0, flags 0x400001002 00010000:00010000:2.0:1734016145.764801:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d2a6 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.764809:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d2a6 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.765620:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d291 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786bc expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.765635:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016145.765650:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332537472:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016145.765880:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d291 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786bc expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.765894:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d291 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786bc expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.765903:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d291 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786bc expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.773092:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334642432 00000100:00100000:2.0:1734016145.773111:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334642432 00000100:00100000:2.0:1734016145.773200:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14747 00000100:00100000:2.0:1734016145.773282:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334642432:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016145.773311:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.773328:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.773377:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 122, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016145.773382:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4682200 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 120 00000020:00010000:2.0:1734016145.773385:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016145.773386:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4682200 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 120 00000100:00100000:2.0:1734016145.773481:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334642432:12345-192.168.201.152@tcp:106: Request processed in 270us (721us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.773489:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14747 00010000:00010000:2.0:1734016145.783351:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d2a6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.783393:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d2a6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.783419:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d2a6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.783426:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d2a6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.783442:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d2a6 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.783475:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4682200, setting rss=4682200; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016145.783504:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d2a6 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.783512:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d2a6 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.783520:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d2a6 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.783536:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbdd40 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016145.783543:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d2a6 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.783557:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016145.783567:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d2a6 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.783574:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d2a6 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.783639:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332537472:192.168.201.152@tcp:101:fsx.0 00000080:00200000:2.0:1734016145.796293:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8876504 i_blocks 136 00000020:00010000:2.0:1734016145.796304:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016145.796310:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.796317:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676030 (ffff8800a72d9098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.796327:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676070 (ffff8800a72d9878/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.796332:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.796334:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676030 (ffff8800a72d9bd8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.796343:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676070 (ffff8800a72d9bd8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016145.796362:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016145.796486:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 507 00000080:00200000:2.0:1734016145.796502:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016145.796519:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016145.796550:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [507, 507] 00000008:00010000:2.0:1734016145.796575:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016145.796581:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef81b0, osc lock ffff8800aafceab0, flags 0x0 00010000:00010000:2.0:1734016145.796610:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.796624:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2076672 2080767) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.797106:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.797114:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.797121:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.797269:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:2.0:1734016145.797285:0:1970:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.797322:0:1970:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2076672 2080767) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016145.797373:0:1970:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73c8000 - object 0x280000400:7 offset 2076672<>2080768 00000100:00100000:2.0:1734016145.797404:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332537856:192.168.201.152@tcp:3:fsx.0 00000008:00010000:2.0:1734016145.801555:0:1970:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016145.801589:0:1970:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.801613:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332537856:192.168.201.152@tcp:3:fsx.0 00000080:00200000:2.0:1734016145.818782:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016145.818804:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016145.818817:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1734016145.818896:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016145.818912:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332538112:192.168.201.152@tcp:16:fsx.0 00000100:00100000:2.0:1734016145.831291:0:1970:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8800b3cc4800: committing for last_committed 4294969911 gen 1 00000100:00100000:2.0:1734016145.831345:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332538112:192.168.201.152@tcp:16:fsx.0 00000100:00100000:2.0:1734016145.842117:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334642560 00000100:00100000:2.0:1734016145.842134:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334642560 00000100:00100000:2.0:1734016145.842794:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14748 00000100:00100000:2.0:1734016145.842807:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:5762f609-0e87-4e8c-9bd6-d528092c9d9c+5:7713:x1818247334642560:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016145.842835:0:9532:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c23a80/0x37cab3886ce5d26e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb46129d8d3178699 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016145.843654:0:9532:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016145.843668:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:5762f609-0e87-4e8c-9bd6-d528092c9d9c+5:7713:x1818247334642560:12345-192.168.201.152@tcp:104: Request processed in 861us (2145us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.843680:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14748 00000100:00100000:2.0:1734016145.859407:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016145.860386:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016145.860431:0:4728:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016145.860444:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334642688 00000100:00100000:2.0:1734016145.860453:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334642688 00000100:00100000:2.0:1734016145.860485:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334642816 00000100:00100000:2.0:1734016145.860487:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334642816 00000100:00100000:2.0:1734016145.860496:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14749 00000100:00100000:2.0:1734016145.860502:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73c8000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334642688:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016145.860518:0:4728:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21f80/0x37cab3886ce5d2ad lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0xb46129d8d31786d8 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.860588:0:4728:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016145.860593:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73c8000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334642688:12345-192.168.201.152@tcp:104: Request processed in 91us (1198us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.860598:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14749 00000100:00100000:2.0:1734016145.860615:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14750 00000100:00100000:2.0:1734016145.860619:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7039450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334642816:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016145.860627:0:4728:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d291 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2953216->2961407) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d31786bc expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.860660:0:4728:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016145.860664:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7039450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334642816:12345-192.168.201.152@tcp:104: Request processed in 44us (281us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.860668:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14750 00000100:00100000:2.0:1734016145.871193:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016145.885040:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8510952 i_blocks 128 00000020:00010000:2.0:1734016145.885051:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8e48 (ffff8800a72d9bd8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016145.885055:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8e48 (ffff8800a72d9bd8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.885061:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676930 (ffff8800a72d9878/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016145.885083:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.885100:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.885121:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676970 (ffff8800a72d92d8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016145.885127:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.885135:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.885145:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8e48 (ffff8800a72d9bd8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.885147:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676930 (ffff8800a72d9bd8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.885154:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676970 (ffff8800a72d9bd8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016145.885178:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016145.885197:0:10366:0:(llite_lib.c:2534:ll_setattr_raw()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8) current size 8510952, valid attrs 2168, mode 8800, uid -1451891104, gid -30720, new size 8510952, atime 1734016145.000000000, mtime 1734016145.831000000, ctime 1734016145.831000000, rc 0 00000080:00200008:2.0:1734016145.885263:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016145.885270:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016145.885295:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016145.885318:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b62b8800 type 13 mode 30 res 8589935617/12 (0 0) 00010000:00010000:2.0:1734016145.885394:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016145.885402:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00010000:00010000:2.0:1734016145.885411:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 3/1,0 mode: --/CR res: [0x200000401:0xc:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 10366 initiator: MDT0 00000100:00100000:2.0:1734016145.885436:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332539264:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016145.885509:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e26c0 going to sleep for 11 seconds 00000100:00100000:2.0:1734016145.934910:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016145.941658:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016145.971425:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016145.971560:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016145.971620:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334643328 00000100:00100000:2.0:1734016145.971634:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334643328 00000100:00100000:2.0:1734016145.971683:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334643456 00000100:00100000:2.0:1734016145.971686:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334643456 00000100:00100000:2.0:1734016145.971699:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14754 00000100:00100000:2.0:1734016145.971708:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73cb800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334643328:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016145.971728:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.971741:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.971783:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 33, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016145.971786:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 16 00000020:00010000:2.0:1734016145.971791:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016145.971792:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 16 00000100:00100000:2.0:1734016145.971857:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73cb800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334643328:12345-192.168.201.152@tcp:106: Request processed in 150us (445us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.971864:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14754 00000100:00100000:2.0:1734016145.971892:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14755 00000100:00100000:2.0:1734016145.971896:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73c8a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334643456:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016145.971904:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.971911:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.971924:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 33, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016145.971926:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 112 00000020:00010000:2.0:1734016145.971929:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016145.971930:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 112 00000100:00100000:2.0:1734016145.971948:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73c8a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334643456:12345-192.168.201.152@tcp:106: Request processed in 50us (388us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016145.971956:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14755 00000020:00010000:2.0:1734016145.998445:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8b78 (ffff8800a72d95a8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016145.998451:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8b78 (ffff8800a72d95a8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016145.998456:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f80b0 (ffff88012db7cea8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016145.998469:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.998483:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016145.998500:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8b78 (ffff8800a72d95a8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016145.998503:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f80b0 (ffff8800a72d95a8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016145.998568:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016145.998668:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016145.998676:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016145.998702:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016145.998720:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016145.998731:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016145.998744:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016145.998747:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d2c2 00000080:00010000:2.0:1734016145.998752:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2c2 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016145.998755:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016145.998761:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016145.998779:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016145.998789:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016145.998812:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016145.998815:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e26f0, osc lock ffff8800aafcef70, flags 0x400001002 00000008:00010000:2.0:1734016145.998821:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016145.998823:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2730, osc lock ffff8800aafce4c0, flags 0x400001002 00010000:00010000:2.0:1734016145.998848:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d2e5 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.998856:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d2e5 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.998933:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d2ec lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016145.998939:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d2ec lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016145.998980:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332540416:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016145.999039:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016145.999044:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332540288:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016146.003298:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016146.003348:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016146.003369:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334643584 00000100:00100000:2.0:1734016146.003381:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334643584 00000100:00100000:2.0:1734016146.003413:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14756 00000100:00100000:2.0:1734016146.003419:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73cbb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7694:x1818247334643584:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.003431:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.003441:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.003463:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 33, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.003465:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 24 00000020:00010000:2.0:1734016146.003468:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.003469:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 24 00000100:00100000:2.0:1734016146.003529:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73cbb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7694:x1818247334643584:12345-192.168.201.152@tcp:106: Request processed in 111us (239us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.003534:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14756 00010000:00010000:2.0:1734016146.015054:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d2e5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.015090:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d2e5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.015112:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d2e5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.015120:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d2e5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.015137:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d2e5 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.015159:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4316648, setting rss=4316648; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016146.015273:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d2e5 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.015279:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d2e5 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.015288:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d2e5 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.015297:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbc000 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016146.015304:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d2e5 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.015322:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016146.015330:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d2e5 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.015337:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d2e5 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.015392:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332540288:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016146.018788:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334643840 00000100:00100000:2.0:1734016146.018802:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334643840 00000100:00100000:2.0:1734016146.018873:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14758 00000100:00100000:2.0:1734016146.018884:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334643840:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.018904:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.018918:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.018953:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 33, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.018957:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 24 00000020:00010000:2.0:1734016146.018960:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.018961:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 24 00000100:00100000:2.0:1734016146.019019:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334643840:12345-192.168.201.152@tcp:106: Request processed in 136us (354us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.019027:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14758 00000100:00100000:2.0:1734016146.019473:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334643968 00000100:00100000:2.0:1734016146.019478:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334643968 00000100:00100000:2.0:1734016146.019506:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14759 00000100:00100000:2.0:1734016146.019511:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334643968:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.019528:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.019535:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.019554:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 33, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.019556:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 112 00000020:00010000:2.0:1734016146.019558:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.019559:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 112 00000100:00100000:2.0:1734016146.019584:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334643968:12345-192.168.201.152@tcp:106: Request processed in 72us (392us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.019591:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14759 00010000:00010000:2.0:1734016146.021635:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964df80/0x37cab3886ce5d2fa lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.021662:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964df80/0x37cab3886ce5d2fa lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.021680:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964df80/0x37cab3886ce5d2fa lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.021689:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964df80/0x37cab3886ce5d2fa lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.021698:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964df80/0x37cab3886ce5d2fa lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.021722:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016146.021733:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964df80/0x37cab3886ce5d2fa lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.021745:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964df80/0x37cab3886ce5d2fa lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.021753:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964df80/0x37cab3886ce5d2fa lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.021762:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a964df80 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016146.021769:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964df80/0x37cab3886ce5d2fa lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.021786:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016146.021795:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964df80/0x37cab3886ce5d2fa lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.021801:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964df80/0x37cab3886ce5d2fa lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.021843:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332540672:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016146.023361:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d2f3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.023381:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d2f3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.023393:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d2f3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.023399:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d2f3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.023403:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d2f3 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.023421:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4316648, setting rss=4316648; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016146.023441:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d2f3 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.023445:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d2f3 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.023451:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d2f3 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.023458:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a964eac0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016146.023465:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d2f3 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.023475:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016146.023481:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d2f3 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.023486:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d2f3 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.023532:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332540544:192.168.201.152@tcp:101:fsx.0 00000080:00200000:2.0:1734016146.024298:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8510952 i_blocks 136 00000020:00010000:2.0:1734016146.024306:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.024310:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.024316:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a708c930 (ffff8800a72d9878/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.024320:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a708c970 (ffff8800a72d92d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.024323:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.024325:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a708c930 (ffff8800a72d9bd8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.024329:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a708c970 (ffff8800a72d9bd8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.024347:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.024359:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8510952 00000080:00200008:2.0:1734016146.024383:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016146.024392:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016146.024409:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016146.024433:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.024579:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016146.024596:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016146.024600:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d2bb 00000080:00010000:2.0:1734016146.024608:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2bb from it ffff8800a7343cd8 00010000:00010000:2.0:1734016146.024611:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.024619:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016146.024646:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.024657:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.024679:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.024682:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e20f0, osc lock ffff8800aafce850, flags 0x400001002 00000008:00010000:2.0:1734016146.024690:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.024691:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2130, osc lock ffff8800aafcf1d0, flags 0x400001002 00010000:00010000:2.0:1734016146.024713:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.024721:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.024729:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.024734:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.024750:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.024756:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.024768:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.024774:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016146.024790:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8510952 i_blocks 136 00000020:00010000:2.0:1734016146.024793:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.024795:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.024797:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e20f0 (ffff8800a72d92d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.024799:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2130 (ffff8800a72d9878/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.024802:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.024804:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e20f0 (ffff8800a72d9bd8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.024808:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2130 (ffff8800a72d9bd8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.024816:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016146.024840:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:2.0:1734016146.024846:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:2.0:1734016146.024859:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016146.024874:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [2489, 2491] 00000008:00010000:2.0:1734016146.024885:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016146.024887:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8c30, osc lock ffff8800aafcf1d0, flags 0x0 00010000:00010000:2.0:1734016146.024893:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.024900:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (6000640 6012927) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.024912:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.024920:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.024927:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016146.024940:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8510952 i_blocks 136 00000100:00100000:2.0:1734016146.025054:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016146.025079:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332540800:192.168.201.152@tcp:22:fsx.0 00000100:00100000:2.0:1734016146.032732:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332540800:192.168.201.152@tcp:22:fsx.0 00000100:00100000:2.0:1734016146.036381:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334644096 00000100:00100000:2.0:1734016146.036398:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334644096 00000100:00100000:2.0:1734016146.036447:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334644224 00000100:00100000:2.0:1734016146.036449:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334644224 00000100:00100000:2.0:1734016146.036471:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14760 00000100:00100000:2.0:1734016146.036482:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334644096:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.036503:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.036515:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.036569:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 33, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.036571:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 24 00000020:00010000:2.0:1734016146.036574:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.036575:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 24 00000100:00100000:2.0:1734016146.036644:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334644096:12345-192.168.201.152@tcp:106: Request processed in 163us (529us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.036653:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14760 00000100:00100000:2.0:1734016146.036681:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14761 00000100:00100000:2.0:1734016146.036684:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334644224:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.036691:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.036698:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.036711:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 33, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.036712:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 136 00000020:00010000:2.0:1734016146.036715:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.036715:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 136 00000100:00100000:2.0:1734016146.036732:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334644224:12345-192.168.201.152@tcp:106: Request processed in 47us (533us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.036737:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14761 00000080:00200000:2.0:1734016146.043509:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8510952 i_blocks 160 00000020:00010000:2.0:1734016146.043518:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.043534:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.043540:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86763f0 (ffff8800a72d9878/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.043545:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676430 (ffff8800a72d9ab8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.043547:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd488 (ffff8800a72d9bd8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.043549:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86763f0 (ffff8800a72d9bd8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.043554:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676430 (ffff8800a72d9bd8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.043567:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.043630:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016146.043639:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.043650:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.043668:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.043670:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e20f0, osc lock ffff8800aafce850, flags 0x400001002 00000008:00010000:2.0:1734016146.043678:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.043679:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2130, osc lock ffff8800aafcf430, flags 0x400001002 00010000:00010000:2.0:1734016146.043705:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d30f lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.043713:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d30f lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.045077:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d316 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.045087:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d316 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.049463:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334644352 00000100:00100000:2.0:1734016146.049481:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334644352 00000100:00100000:2.0:1734016146.049561:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334644480 00000100:00100000:2.0:1734016146.049564:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334644480 00000100:00100000:2.0:1734016146.049582:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14762 00000100:00100000:2.0:1734016146.049596:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7694:x1818247334644352:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.049622:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.049640:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.049689:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 33, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.049692:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 136 00000020:00010000:2.0:1734016146.049695:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.049697:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 136 00000100:00100000:2.0:1734016146.049788:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7694:x1818247334644352:12345-192.168.201.152@tcp:106: Request processed in 195us (652us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.049796:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14762 00000100:00100000:2.0:1734016146.049825:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14763 00000100:00100000:2.0:1734016146.049829:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341d880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334644480:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.049835:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.049842:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.049858:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 33, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.049859:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 24 00000020:00010000:2.0:1734016146.049863:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.049865:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016145 atime: 1734016145 ctime: 1734016145 blocks: 24 00000100:00100000:2.0:1734016146.049882:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341d880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334644480:12345-192.168.201.152@tcp:106: Request processed in 53us (553us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.049889:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14763 00010000:00010000:2.0:1734016146.056316:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d316 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.056327:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d316 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.056335:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d316 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.056342:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d316 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.056347:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d316 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.056357:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016146.056373:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d316 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.056379:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d316 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.056384:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d316 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.056396:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbf600 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016146.056399:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d316 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.056407:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016146.056410:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d316 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.056416:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d316 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.056438:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332541312:192.168.201.152@tcp:101:fsx.0 00000020:00010000:2.0:1734016146.064367:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.064372:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.064377:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f87b0 (ffff8800a72d9878/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016146.064391:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.064404:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.064424:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.064427:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f87b0 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.064451:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016146.064476:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016146.064497:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016146.064515:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [624, 624] 00000008:00010000:2.0:1734016146.064544:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016146.064551:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8cb0, osc lock ffff8800aafce130, flags 0x0 00010000:00010000:2.0:1734016146.064577:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.064593:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2555904 2559999) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.064604:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.064611:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.064620:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.064673:0:10366:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016146.064700:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 1, dirtied 1 00000020:00010000:2.0:1734016146.064717:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.064720:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.064722:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8cb0 (ffff8800a72d9878/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016146.064725:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.064738:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.064746:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.064748:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8cb0 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.064760:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016146.064835:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 625 00000080:00200000:2.0:1734016146.064842:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016146.064847:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016146.064852:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [625, 625] 00000008:00010000:2.0:1734016146.064856:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016146.064858:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8cb0, osc lock ffff8800aafce130, flags 0x0 00010000:00010000:2.0:1734016146.064864:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.064873:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2560000 2564095) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.064881:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.064888:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.064893:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.064983:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:2.0:1734016146.064994:0:1970:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e2f00 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.065023:0:1970:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2560000 2564095) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016146.065063:0:1970:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73c8380 - object 0x280000400:7 offset 2560000<>2564096 00000100:00100000:2.0:1734016146.065086:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332541824:192.168.201.152@tcp:3:fsx.0 00000008:00010000:2.0:1734016146.068832:0:1970:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e2f00 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.068883:0:1970:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2f00 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.068918:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332541824:192.168.201.152@tcp:3:fsx.0 00000020:00010000:2.0:1734016146.074796:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.074801:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.074804:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f8430 (ffff8800a72d9878/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016146.074813:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.074825:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.074850:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.074853:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f8430 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.074874:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016146.074894:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016146.074910:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016146.074930:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [626, 626] 00000008:00010000:2.0:1734016146.074947:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016146.074953:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8730, osc lock ffff8800aafce000, flags 0x0 00010000:00010000:2.0:1734016146.074978:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.074992:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2564096 2568191) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.075005:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.075012:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.075017:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.075063:0:10366:0:(osc_cache.c:543:osc_extent_merge()) ### extent: ffff8800aa7e2f00 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.075069:0:10366:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2f00 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.075076:0:10366:0:(osc_cache.c:560:osc_extent_merge()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.075086:0:10366:0:(osc_cache.c:771:osc_extent_find()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016146.075104:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 1, dirtied 1 00000020:00010000:2.0:1734016146.075114:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.075115:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.075117:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8730 (ffff8800a72d9878/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016146.075119:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.075124:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.075131:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.075132:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8730 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.075144:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016146.075181:0:10366:0:(file.c:5185:ll_fsync()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8), start 2555904, end 2568191, datasync 1 00000080:00200000:2.0:1734016146.075189:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016146.075193:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1734016146.075206:0:10366:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.075254:0:10366:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.075428:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2555904 2559999) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016146.075539:0:10366:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73c8380 - object 0x280000400:7 offset 2555904<>2568192 00000100:00100000:2.0:1734016146.075700:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016146.075721:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332542208:192.168.201.152@tcp:4:fsx.0 00000008:00010000:2.0:1734016146.080299:0:1970:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.080372:0:1970:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.080395:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332542208:192.168.201.152@tcp:4:fsx.0 00000080:00200000:2.0:1734016146.092132:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8510952 i_blocks 184 00000080:00200000:2.0:1734016146.092155:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.092376:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016146.092383:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016146.092403:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016146.092423:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.092438:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016146.092453:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016146.092458:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d2bb 00000080:00010000:2.0:1734016146.092469:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2bb from it ffff8800a7343cd8 00010000:00010000:2.0:1734016146.092473:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.092481:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016146.092525:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.092535:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.092552:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.092556:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f30, osc lock ffff8800aafce000, flags 0x400001002 00000008:00010000:2.0:1734016146.092563:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.092564:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f70, osc lock ffff8800aafce260, flags 0x400001002 00010000:00010000:2.0:1734016146.092579:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.092587:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.092599:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.092605:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.092620:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.092635:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.092642:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.092647:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016146.092660:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8510952 i_blocks 184 00000020:00010000:2.0:1734016146.092664:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.092667:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.092670:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f30 (ffff8800a72d9ea8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.092674:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f70 (ffff8800a72d9098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.092677:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.092679:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f30 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.092684:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f70 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.092690:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.092708:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016146.092712:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.092716:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.092720:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.092721:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f30, osc lock ffff8800aafce260, flags 0x400001002 00000008:00010000:2.0:1734016146.092724:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.092726:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f70, osc lock ffff8800aafce000, flags 0x400001002 00010000:00010000:2.0:1734016146.092730:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.092738:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.092746:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.092753:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.092762:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.092767:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.092774:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.092780:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016146.092787:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8510952 i_blocks 184 00000020:00010000:2.0:1734016146.092790:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.092793:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.092795:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f30 (ffff8800a72d9098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.092799:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f70 (ffff8800a72d9ea8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.092801:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.092803:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f30 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.092806:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f70 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.092810:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.092817:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8510952 00000080:00200008:2.0:1734016146.092831:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016146.092838:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016146.092849:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016146.092862:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.092871:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016146.092878:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016146.092880:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d2c2 00000080:00010000:2.0:1734016146.092883:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2c2 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016146.092886:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.092892:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016146.092907:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.092911:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.092919:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.092920:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f30, osc lock ffff8800aafce000, flags 0x400001002 00000008:00010000:2.0:1734016146.092925:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.092927:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f70, osc lock ffff8800aafce260, flags 0x400001002 00010000:00010000:2.0:1734016146.092954:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd440/0x37cab3886ce5d31d lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.092962:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd440/0x37cab3886ce5d31d lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.093048:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d324 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.093057:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d324 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.097270:0:4729:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016146.097294:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334644608 00000100:00100000:2.0:1734016146.097302:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334644608 00000100:00100000:2.0:1734016146.097357:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14764 00000100:00100000:2.0:1734016146.097367:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012b52e450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334644608:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.097385:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.097401:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.097437:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 36, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.097440:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 160 00000020:00010000:2.0:1734016146.097443:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.097444:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 160 00000100:00100000:2.0:1734016146.097505:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012b52e450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334644608:12345-192.168.201.152@tcp:106: Request processed in 140us (686us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.097512:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14764 00010000:00010000:2.0:1734016146.103028:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d324 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.103053:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d324 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.103071:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d324 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.103076:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d324 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.103084:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d324 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.103100:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016146.103109:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d324 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.103114:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d324 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.103122:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d324 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.103134:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbeac0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016146.103139:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d324 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.103152:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016146.103157:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d324 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.103161:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d324 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.103253:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332542720:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016146.103273:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd440/0x37cab3886ce5d31d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.103281:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd440/0x37cab3886ce5d31d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.103289:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd440/0x37cab3886ce5d31d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.103295:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd440/0x37cab3886ce5d31d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.103300:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd440/0x37cab3886ce5d31d lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.103313:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4316648, setting rss=4316648; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016146.103829:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd440/0x37cab3886ce5d31d lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.103837:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd440/0x37cab3886ce5d31d lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.103843:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd440/0x37cab3886ce5d31d lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.103851:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbd440 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016146.103855:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd440/0x37cab3886ce5d31d lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.103864:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016146.103867:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd440/0x37cab3886ce5d31d lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.103873:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd440/0x37cab3886ce5d31d lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.103892:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332542592:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016146.109428:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334644864 00000100:00100000:2.0:1734016146.109440:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334644864 00000100:00100000:2.0:1734016146.109507:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14765 00000100:00100000:2.0:1734016146.109528:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ca80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334644864:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016146.109551:0:9532:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a964ed00/0x37cab3886ce5d24b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4677632->4685823) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d317866f expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.110206:0:9532:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016146.110214:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ca80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334644864:12345-192.168.201.152@tcp:104: Request processed in 695us (916us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.110268:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14765 00000100:00100000:2.0:1734016146.116066:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000020:00010000:2.0:1734016146.122764:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.122770:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.122778:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b35bacb0 (ffff8800a72d9098/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016146.122796:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.122815:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.122837:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.122840:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b35bacb0 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.122869:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.122971:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016146.122983:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016146.123025:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016146.123058:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.123083:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016146.123105:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016146.123109:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d2c2 00000080:00010000:2.0:1734016146.123116:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2c2 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016146.123119:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.123125:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016146.123150:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.123169:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.123198:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.123204:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2630, osc lock ffff8800aafcebe0, flags 0x400001002 00000008:00010000:2.0:1734016146.123212:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.123214:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2670, osc lock ffff8800aafce390, flags 0x400001002 00010000:00010000:2.0:1734016146.123269:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.123279:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.123293:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.123300:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.123334:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d332 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.123342:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d332 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.129072:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334645120 00000100:00100000:2.0:1734016146.129090:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334645120 00000100:00100000:2.0:1734016146.129143:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14767 00000100:00100000:2.0:1734016146.129151:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334645120:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.129172:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.129185:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.129304:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.129307:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 24 00000020:00010000:2.0:1734016146.129310:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.129311:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 24 00000100:00100000:2.0:1734016146.129374:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334645120:12345-192.168.201.152@tcp:106: Request processed in 224us (447us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.129381:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14767 00010000:00010000:2.0:1734016146.131679:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d332 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.131708:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d332 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.131720:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d332 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.131726:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d332 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.131734:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d332 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.131751:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016146.131767:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d332 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.131774:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d332 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.131781:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d332 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.131786:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbd200 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016146.131792:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d332 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.131809:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016146.131817:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d332 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.131824:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d332 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.131880:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332543232:192.168.201.152@tcp:101:fsx.0 00000080:00200000:2.0:1734016146.142367:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8510952 i_blocks 200 00000020:00010000:2.0:1734016146.142374:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.142377:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.142382:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676cf0 (ffff88012db7c7e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.142386:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676d30 (ffff8800a72d9098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.142388:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.142390:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676cf0 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.142394:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676d30 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.142407:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.142418:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8510952 00000080:00200008:2.0:1734016146.142489:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 5740157, type: SEEK_SET 00000080:00200008:2.0:1734016146.142497:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 5740157, type: SEEK_SET, rc = 5740157 00000080:00200008:2.0:1734016146.142506:0:10366:0:(file.c:2466:do_file_write_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 5740157, count: 2952 00000080:00200000:2.0:1734016146.142513:0:10366:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: write ppos: 5740157, bytes: 2952 00000080:00200000:2.0:1734016146.142526:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:2.0:1734016146.142531:0:10366:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b62c6a28[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00200000:2.0:1734016146.142533:0:10366:0:(cl_io.c:195:cl_io_rw_init()) io range: 2 [5740157, 5743109) 0 0 00000080:00200000:2.0:1734016146.142538:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016146.142553:0:10366:0:(file.c:1893:ll_file_io_generic()) Range lock [1401, 1402] 00020000:00200000:2.0:1734016146.142560:0:10366:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 1 chunk: [5740157, 5743109) 5743109, 2952 00020000:00200000:2.0:1734016146.142564:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016146.142582:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1401, 1402] 00000008:00010000:2.0:1734016146.142601:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016146.142606:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8bb0, osc lock ffff8800aafce390, flags 0x0 00010000:00010000:2.0:1734016146.142628:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.142640:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1544192 1552383) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.142648:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.142653:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.142658:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016146.142665:0:10366:0:(vvp_io.c:1326:vvp_io_write_start()) f16b.sanityn: write [5740157, 5743109) 00000080:00200000:2.0:1734016146.142691:0:10366:0:(rw26.c:819:ll_write_begin()) Writing 1401 of 1661 to 2435 bytes 00000008:00010000:2.0:1734016146.147008:0:1970:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e2b40 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.147810:0:1970:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2b40 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.147847:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332543616:192.168.201.152@tcp:3:fsx.0 00000080:00200000:2.0:1734016146.152869:0:10366:0:(rw26.c:1006:ll_write_end()) pos 5742592, len 517, copied 517 00000080:00200000:2.0:1734016146.152879:0:10366:0:(rw26.c:1044:ll_write_end()) page@ffff8800a70cc870[3 ffff8800b62c6aa8 2 1 ffff8800a70dd2a0] 00000080:00200000:2.0:1734016146.152881:0:10366:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002a27280 1fffff0000080d 3:0 ffff8800a70cc870 1402 lru 00000080:00200000:2.0:1734016146.152888:0:10366:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800a70cc8d0 378: 1< 1 - - > 2< 1548288 0 4096 0x0 0x40009 | ffff8800a97ac758 ffff88012b46ab60 > 3< 0 0 > 4< 0 0 8 289820672 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1734016146.152890:0:10366:0:(rw26.c:1044:ll_write_end()) end page@ffff8800a70cc870 00000080:00200000:2.0:1734016146.152892:0:10366:0:(rw26.c:1044:ll_write_end()) queued page: 2. 00000080:00200000:2.0:1734016146.152903:0:10366:0:(vvp_io.c:1218:vvp_io_write_commit()) commit async pages: 2, from 1661, to 517 00000008:00010000:2.0:1734016146.152931:0:10366:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800aa7e2b40 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.152951:0:10366:0:(osc_cache.c:2323:osc_queue_async_io()) ### extent: ffff8800aa7e2b40 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016146.152965:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 2, dirtied 2 00000080:00200000:2.0:1734016146.152970:0:10366:0:(vvp_io.c:1241:vvp_io_write_commit()) Committed 2 pages 2952 bytes, tot: 2952 00000080:00200000:2.0:1734016146.152973:0:10366:0:(vvp_io.c:1405:vvp_io_write_start()) f16b.sanityn: write bytes 0, result: 2952 00000020:00010000:2.0:1734016146.152988:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.152991:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.152995:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8bb0 (ffff8800a72d9098/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016146.153002:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.153010:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.153031:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.153034:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8bb0 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.153043:0:10366:0:(vvp_io.c:533:vvp_io_advance()) advancing 2952 bytes 00000080:00200000:2.0:1734016146.153049:0:10366:0:(file.c:1930:ll_file_io_generic()) Range unlock [1401, 1402] 00000080:00200000:2.0:1734016146.153056:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016146.153059:0:10366:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 2 io complete with rc: 0, result: 2952, restart: 0 00000080:00200000:2.0:1734016146.153068:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8510952 i_blocks 184 00000080:00200000:2.0:1734016146.153073:0:10366:0:(file.c:2044:ll_file_io_generic()) iot: 2, result: 2952 00000080:00200008:2.0:1734016146.153099:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016146.153109:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016146.153127:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016146.153142:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.153150:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016146.153163:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016146.153166:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d2bb 00000080:00010000:2.0:1734016146.153170:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2bb from it ffff8800a7343cd8 00010000:00010000:2.0:1734016146.153173:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.153179:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016146.153193:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.153205:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.153238:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.153241:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2270, osc lock ffff8800aafce390, flags 0x400001002 00000008:00010000:2.0:1734016146.153246:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.153248:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e22b0, osc lock ffff8800aafcebe0, flags 0x400001002 00010000:00010000:2.0:1734016146.153265:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d340 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.153272:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d340 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.153876:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.153891:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016146.153912:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332544000:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016146.154137:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.154149:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.154156:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.156380:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334645376 00000100:00100000:2.0:1734016146.156392:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334645376 00000100:00100000:2.0:1734016146.156451:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14769 00000100:00100000:2.0:1734016146.156460:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334645376:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.156479:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.156490:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.156535:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.156537:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 176 00000020:00010000:2.0:1734016146.156539:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.156541:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 176 00000100:00100000:2.0:1734016146.156592:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334645376:12345-192.168.201.152@tcp:106: Request processed in 132us (704us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.156599:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14769 00010000:00010000:2.0:1734016146.159712:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d340 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.159736:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d340 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.159750:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d340 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.159757:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d340 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.159764:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d340 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.159787:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4316648, setting rss=4316648; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016146.159954:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d340 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.159961:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d340 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.159967:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d340 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.159973:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbd200 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016146.159979:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d340 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.159990:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016146.159995:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d340 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.160002:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d340 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.160530:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332544000:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016146.163664:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334645504 00000100:00100000:2.0:1734016146.163673:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334645504 00000100:00100000:2.0:1734016146.163718:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14770 00000100:00100000:2.0:1734016146.163726:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334645504:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.163742:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.163755:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.163782:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.163784:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 176 00000020:00010000:2.0:1734016146.163787:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.163788:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 176 00000100:00100000:2.0:1734016146.163830:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334645504:12345-192.168.201.152@tcp:106: Request processed in 104us (321us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.163836:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14770 00000080:00200000:2.0:1734016146.167566:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8510952 i_blocks 200 00000020:00010000:2.0:1734016146.167575:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.167578:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.167584:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676270 (ffff88012db7c7e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.167587:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86762b0 (ffff8800a72d9098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.167591:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.167593:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676270 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.167597:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86762b0 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.167614:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.167629:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8510952 00000080:00200008:2.0:1734016146.167656:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016146.167665:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016146.167689:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016146.167710:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.167720:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016146.167736:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016146.167741:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d2bb 00000080:00010000:2.0:1734016146.167746:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2bb from it ffff8800a7343cd8 00010000:00010000:2.0:1734016146.167750:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.167756:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016146.167779:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.167790:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.167808:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.167810:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2870, osc lock ffff8800aafce260, flags 0x400001002 00000008:00010000:2.0:1734016146.167815:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.167816:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e28b0, osc lock ffff8800aafce000, flags 0x400001002 00010000:00010000:2.0:1734016146.167836:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d34e lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.167842:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d34e lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.167911:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.167924:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.167933:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.167939:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.172206:0:4729:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016146.172265:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334645632 00000100:00100000:2.0:1734016146.172276:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334645632 00000100:00100000:2.0:1734016146.172327:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14771 00000100:00100000:2.0:1734016146.172336:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a703ac50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334645632:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.172356:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.172367:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.172398:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.172402:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 176 00000020:00010000:2.0:1734016146.172405:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.172405:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 176 00000100:00100000:2.0:1734016146.172464:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a703ac50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334645632:12345-192.168.201.152@tcp:106: Request processed in 128us (589us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.172473:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14771 00010000:00010000:2.0:1734016146.174457:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d34e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.174469:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d34e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.174479:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d34e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.174484:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d34e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.174490:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d34e lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.174506:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4316648, setting rss=4316648; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016146.174524:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d34e lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.174529:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d34e lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.174535:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d34e lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.174540:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbd200 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016146.174545:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d34e lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.174554:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016146.174558:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d34e lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.174563:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd200/0x37cab3886ce5d34e lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.174597:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332544256:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016146.179399:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334645760 00000100:00100000:2.0:1734016146.179414:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334645760 00000100:00100000:2.0:1734016146.179490:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14772 00000100:00100000:2.0:1734016146.179502:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334645760:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.179532:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.179549:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.179601:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.179604:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 176 00000020:00010000:2.0:1734016146.179607:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.179608:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 176 00000100:00100000:2.0:1734016146.179685:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334645760:12345-192.168.201.152@tcp:106: Request processed in 184us (501us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.179700:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14772 00000080:00200000:2.0:1734016146.182671:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8510952 i_blocks 200 00000020:00010000:2.0:1734016146.182683:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.182688:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.182695:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676330 (ffff88012db7c7e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.182701:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676370 (ffff8800a72d9098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.182707:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.182709:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676330 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.182716:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676370 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.182738:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.182756:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8510952 00000080:00200008:2.0:1734016146.182783:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016146.182794:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016146.182829:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016146.182854:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.182867:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016146.182889:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016146.182893:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d2bb 00000080:00010000:2.0:1734016146.182902:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2bb from it ffff8800a7343cd8 00010000:00010000:2.0:1734016146.182905:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.182916:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016146.182948:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.182963:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.182990:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.182993:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e23f0, osc lock ffff8800aafce130, flags 0x400001002 00000008:00010000:2.0:1734016146.183002:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.183003:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2430, osc lock ffff8800aafcf430, flags 0x400001002 00010000:00010000:2.0:1734016146.183039:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d35c lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.183046:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d35c lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.183145:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.183156:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.183170:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.183178:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.186680:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334645888 00000100:00100000:2.0:1734016146.186693:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334645888 00000100:00100000:2.0:1734016146.186739:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14773 00000100:00100000:2.0:1734016146.186745:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334645888:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.186761:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.186771:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.186793:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.186795:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 176 00000020:00010000:2.0:1734016146.186798:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.186799:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 176 00000100:00100000:2.0:1734016146.186846:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334645888:12345-192.168.201.152@tcp:106: Request processed in 101us (754us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.186853:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14773 00010000:00010000:2.0:1734016146.190015:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d35c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.190038:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d35c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.190051:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d35c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.190056:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d35c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.190063:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d35c lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.190086:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4316648, setting rss=4316648; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016146.190110:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d35c lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.190116:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d35c lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.190122:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d35c lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.190128:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbd440 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016146.190133:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d35c lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.190142:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016146.190146:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d35c lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.190151:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d35c lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.190186:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332544512:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016146.193922:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334646016 00000100:00100000:2.0:1734016146.193934:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334646016 00000100:00100000:2.0:1734016146.194000:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14774 00000100:00100000:2.0:1734016146.194009:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334646016:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.194031:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.194045:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.194081:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.194083:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 176 00000020:00010000:2.0:1734016146.194086:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.194087:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 176 00000100:00100000:2.0:1734016146.194148:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334646016:12345-192.168.201.152@tcp:106: Request processed in 140us (389us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.194158:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14774 00000080:00200000:2.0:1734016146.197488:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 8510952 i_blocks 200 00000020:00010000:2.0:1734016146.197498:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.197502:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.197507:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676db0 (ffff88012db7c7e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.197510:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676df0 (ffff8800a72d9098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.197514:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.197524:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676db0 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.197528:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676df0 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.197540:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.197554:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 8510952 00000080:00200008:2.0:1734016146.197575:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016146.197582:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016146.197603:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016146.197620:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.197631:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016146.197644:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016146.197647:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d2bb 00000080:00010000:2.0:1734016146.197653:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2bb from it ffff8800a7343cd8 00010000:00010000:2.0:1734016146.197655:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.197661:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016146.197688:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.197699:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.197718:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.197720:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2870, osc lock ffff8800aafce850, flags 0x400001002 00000008:00010000:2.0:1734016146.197728:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.197729:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e28b0, osc lock ffff8800aafcf1d0, flags 0x400001002 00010000:00010000:2.0:1734016146.197749:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d36a lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.197756:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d36a lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.199009:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.199024:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.199034:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.199040:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.201530:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334646144 00000100:00100000:2.0:1734016146.201541:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334646144 00000100:00100000:2.0:1734016146.201606:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14775 00000100:00100000:2.0:1734016146.201613:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341d880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334646144:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.201629:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.201638:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.201671:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.201673:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 176 00000020:00010000:2.0:1734016146.201676:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.201678:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 176 00000100:00100000:2.0:1734016146.201735:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341d880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334646144:12345-192.168.201.152@tcp:106: Request processed in 121us (417us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.201744:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14775 00010000:00010000:2.0:1734016146.204342:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d36a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.204376:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d36a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.204393:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d36a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.204399:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d36a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.204406:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d36a lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.204428:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4316648, setting rss=4316648; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016146.204985:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d36a lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.204992:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d36a lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.205000:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d36a lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.205008:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbd440 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016146.205014:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d36a lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.205025:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016146.205032:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d36a lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.205039:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d36a lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.205106:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332544768:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016146.208989:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334646272 00000100:00100000:2.0:1734016146.209001:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334646272 00000100:00100000:2.0:1734016146.209105:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14776 00000100:00100000:2.0:1734016146.209115:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334646272:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016146.209137:0:9532:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964eac0/0x37cab3886ce5d32b lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5722112->5730303) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d317876b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.209374:0:9532:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016146.209382:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334646272:12345-192.168.201.152@tcp:104: Request processed in 267us (482us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.209389:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14776 00000100:00100000:2.0:1734016146.211692:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334646400 00000100:00100000:2.0:1734016146.211702:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334646400 00000100:00100000:2.0:1734016146.211749:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14777 00000100:00100000:2.0:1734016146.211757:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6472:x1818247334646400:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016146.211774:0:9532:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c22400/0x37cab3886ce5d371 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1449984->1454079] (req 1449984->1454079) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31787b1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.211785:0:9532:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c22400/0x37cab3886ce5d371 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1449984->1454079] (req 1449984->1454079) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31787b1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.211795:0:9532:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c22400/0x37cab3886ce5d371 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1449984->1454079) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31787b1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.211804:0:9532:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c22400/0x37cab3886ce5d371 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1449984->1454079) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31787b1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.211811:0:9532:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c22400/0x37cab3886ce5d371 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1449984->1454079) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31787b1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.211820:0:9532:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c22400/0x37cab3886ce5d371 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1449984->1454079) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d31787b1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.211854:0:9532:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a6c22400) 00000100:00100000:2.0:1734016146.211904:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6472:x1818247334646400:12345-192.168.201.152@tcp:105: Request processed in 148us (307us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.211911:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14777 00000100:00100000:2.0:1734016146.217653:0:4729:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016146.217673:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334646656 00000100:00100000:2.0:1734016146.217682:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334646656 00000100:00100000:2.0:1734016146.217780:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14778 00000100:00100000:2.0:1734016146.217789:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7039850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334646656:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.217804:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c22400/0x37cab3886ce5d371 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1449984->1454079) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787b1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.217814:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c22400/0x37cab3886ce5d371 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1449984->1454079) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787b1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.217836:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.217838:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 184 00000020:00010000:2.0:1734016146.217841:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.217842:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 184 00000100:00100000:2.0:1734016146.217970:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7039850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334646656:12345-192.168.201.152@tcp:106: Request processed in 182us (409us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.217975:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14778 00000100:00100000:2.0:1734016146.225169:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334646784 00000100:00100000:2.0:1734016146.225189:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334646784 00000100:00100000:2.0:1734016146.225271:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14779 00000100:00100000:2.0:1734016146.225279:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334646784:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.225297:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c22400/0x37cab3886ce5d371 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1449984->1454079) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787b1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.225307:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c22400/0x37cab3886ce5d371 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1449984->1454079) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787b1 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.225342:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.225345:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 184 00000020:00010000:2.0:1734016146.225347:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.225348:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4316648 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 184 00000100:00100000:2.0:1734016146.225399:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7694:x1818247334646784:12345-192.168.201.152@tcp:106: Request processed in 122us (516us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.225405:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14779 00000100:00100000:2.0:1734016146.226339:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016146.246195:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016146.246291:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016146.255071:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016146.271343:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016146.283792:0:4728:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016146.283837:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334647552 00000100:00100000:2.0:1734016146.283858:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334647552 00000100:00100000:2.0:1734016146.283989:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14785 00000100:00100000:2.0:1734016146.284001:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a703bc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334647552:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.284031:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b716d00/0x37cab3886ce5d3a2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5423104->5431295) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787e2 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.284049:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b716d00/0x37cab3886ce5d3a2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5423104->5431295) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787e2 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.284095:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.284098:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 224 00000020:00010000:2.0:1734016146.284101:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.284103:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 224 00000100:00100000:2.0:1734016146.284186:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a703bc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334647552:12345-192.168.201.152@tcp:106: Request processed in 187us (780us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.284194:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14785 00000080:00200000:2.0:1734016146.289359:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 248 00000020:00010000:2.0:1734016146.289368:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.289373:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.289378:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40a7b0 (ffff8800a72d9098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.289382:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40a7f0 (ffff88012db7c7e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.289385:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.289387:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40a7b0 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.289390:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40a7f0 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.289401:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.289425:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016146.289435:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.289446:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.289465:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.289467:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2330, osc lock ffff8800aafce4c0, flags 0x400001002 00000008:00010000:2.0:1734016146.289473:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.289474:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2370, osc lock ffff8800aafcef70, flags 0x400001002 00010000:00010000:2.0:1734016146.289493:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d3b0 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.289521:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d3b0 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.290171:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.290188:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016146.290199:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332547200:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016146.290483:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.290502:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.290514:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.294330:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334647680 00000100:00100000:2.0:1734016146.294347:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334647680 00000100:00100000:2.0:1734016146.294417:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14786 00000100:00100000:2.0:1734016146.294427:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334647680:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.294446:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b716d00/0x37cab3886ce5d3a2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5423104->5431295) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787e2 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.294460:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b716d00/0x37cab3886ce5d3a2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5423104->5431295) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787e2 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.294494:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.294497:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 224 00000020:00010000:2.0:1734016146.294501:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.294502:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 224 00000100:00100000:2.0:1734016146.294577:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334647680:12345-192.168.201.152@tcp:106: Request processed in 150us (419us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.294585:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14786 00010000:00010000:2.0:1734016146.297928:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d3b0 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.297967:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d3b0 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.297986:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d3b0 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.297995:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d3b0 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.298004:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d3b0 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.298027:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5431148, setting rss=5431148; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016146.298053:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d3b0 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.298061:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d3b0 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.298070:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d3b0 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.298080:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbeac0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016146.298088:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d3b0 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.298113:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016146.298120:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d3b0 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.298128:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d3b0 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.298167:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332547200:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016146.304267:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334647808 00000100:00100000:2.0:1734016146.304287:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334647808 00000100:00100000:2.0:1734016146.304346:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14787 00000100:00100000:2.0:1734016146.304354:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334647808:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.304375:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b716d00/0x37cab3886ce5d3a2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5423104->5431295) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787e2 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.304390:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b716d00/0x37cab3886ce5d3a2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5423104->5431295) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787e2 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.304427:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.304430:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 224 00000020:00010000:2.0:1734016146.304432:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.304434:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 224 00000100:00100000:2.0:1734016146.304503:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334647808:12345-192.168.201.152@tcp:106: Request processed in 149us (562us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.304510:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14787 00000080:00200000:2.0:1734016146.312329:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 248 00000020:00010000:2.0:1734016146.312341:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.312345:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.312351:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86766f0 (ffff8800a72d9098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.312356:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676730 (ffff88012db7c7e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.312359:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.312361:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86766f0 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.312368:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676730 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.312385:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016146.312473:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 456 00000080:00200000:2.0:1734016146.312484:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016146.312499:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016146.312525:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [456, 456] 00000008:00010000:2.0:1734016146.312538:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016146.312543:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8bb0, osc lock ffff8800aafceab0, flags 0x0 00010000:00010000:2.0:1734016146.312578:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.312589:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1867776->1871871] (req 1867776->1871871) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.312624:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1867776->1871871] (req 1867776->1871871) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.312651:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332547456:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016146.312763:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e29c0 going to sleep for 11 seconds 00000100:00100000:2.0:1734016146.319269:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334647936 00000100:00100000:2.0:1734016146.319283:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334647936 00000100:00100000:2.0:1734016146.319657:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14788 00000100:00100000:2.0:1734016146.319670:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334647936:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016146.319698:0:4728:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012b716d00/0x37cab3886ce5d3a2 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 5423104->5431295) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d31787e2 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.321060:0:4728:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016146.321079:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334647936:12345-192.168.201.152@tcp:104: Request processed in 1405us (2638us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.321091:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14788 00000100:00100000:2.0:1734016146.326960:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332547456:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016146.326997:0:10366:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1867776->1871871] (req 1867776->1871871) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.327010:0:10366:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1867776->1871871] (req 1867776->1871871) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.327019:0:10366:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1867776->1871871] (req 1867776->1871871) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.332693:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334648064 00000100:00100000:2.0:1734016146.332703:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334648064 00000100:00100000:2.0:1734016146.332755:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14789 00000100:00100000:2.0:1734016146.332764:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6472:x1818247334648064:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016146.332784:0:4728:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1867776->1871871] (req 1867776->1871871) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.332797:0:4728:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1867776->1871871] (req 1867776->1871871) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.332812:0:4728:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.332825:0:4728:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.332835:0:4728:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.332847:0:4728:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.332886:0:4728:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff88012dbbc6c0) 00000100:00100000:2.0:1734016146.334297:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6472:x1818247334648064:12345-192.168.201.152@tcp:105: Request processed in 1522us (1678us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.334310:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14789 00000020:00010000:2.0:1734016146.388125:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.388144:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.388150:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8bb0 (ffff88012db7c7e8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016146.388166:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.388184:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.388211:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.388240:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8bb0 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.388286:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016146.388315:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016146.388340:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016146.388375:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [456, 456] 00000008:00010000:2.0:1734016146.388404:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016146.388411:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8bb0, osc lock ffff8800aafceab0, flags 0x0 00010000:00010000:2.0:1734016146.388445:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.388466:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1867776 1871871) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.388480:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.388488:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.388495:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.388561:0:10366:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800aa7e29c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016146.388585:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 1, dirtied 1 00000020:00010000:2.0:1734016146.388602:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.388604:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.388607:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8bb0 (ffff88012db7c7e8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016146.388611:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.388620:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.388628:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8038 (ffff8800a72d9ab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.388630:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8bb0 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.388637:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016146.388717:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 457 00000080:00200000:2.0:1734016146.388724:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016146.388730:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016146.388734:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [457, 457] 00000008:00010000:2.0:1734016146.388739:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016146.388742:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8bb0, osc lock ffff8800aafceab0, flags 0x0 00010000:00010000:2.0:1734016146.388749:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.388757:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1871872 1875967) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.388764:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.388770:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.388776:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.388915:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00010000:2.0:1734016146.388936:0:1971:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e2c00 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.388961:0:1971:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1871872 1875967) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016146.389005:0:1971:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73c9880 - object 0x280000400:7 offset 1871872<>1875968 00000100:00100000:2.0:1734016146.389030:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332548096:192.168.201.152@tcp:3:fsx.0 00000008:00010000:2.0:1734016146.398114:0:1971:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e2c00 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.399550:0:1971:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2c00 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.399589:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332548096:192.168.201.152@tcp:3:fsx.0 00000080:00200000:2.0:1734016146.442417:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 264 00000080:00200000:2.0:1734016146.442438:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.442626:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016146.442636:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016146.442667:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016146.442697:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.442718:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016146.442735:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016146.442738:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d2bb 00000080:00010000:2.0:1734016146.442749:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2bb from it ffff8800a7343cd8 00010000:00010000:2.0:1734016146.442753:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.442760:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016146.442794:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.442809:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.442833:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.442837:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c30, osc lock ffff8800aafce5f0, flags 0x400001002 00000008:00010000:2.0:1734016146.442847:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.442848:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c70, osc lock ffff8800aafcf7c0, flags 0x400001002 00010000:00010000:2.0:1734016146.442869:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.442879:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.442890:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.442896:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.442911:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.442918:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.442923:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.442928:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016146.442939:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 264 00000020:00010000:2.0:1734016146.442945:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.442947:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.442949:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2c30 (ffff88012db7c7e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.442952:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2c70 (ffff8800a72d9098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.442954:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.442955:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2c30 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.442960:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2c70 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.442967:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.442979:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016146.442983:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.442987:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.442989:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.442992:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c30, osc lock ffff8800aafcf7c0, flags 0x400001002 00000008:00010000:2.0:1734016146.442994:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.442995:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c70, osc lock ffff8800aafce5f0, flags 0x400001002 00010000:00010000:2.0:1734016146.442999:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.443005:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.443013:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.443020:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.443028:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.443034:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.443041:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.443047:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016146.443057:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 264 00000020:00010000:2.0:1734016146.443060:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.443062:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.443065:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2c30 (ffff8800a72d9098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.443068:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2c70 (ffff88012db7c7e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.443070:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.443071:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2c30 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.443073:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2c70 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.443080:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.443088:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9625452 00000080:00200008:2.0:1734016146.443098:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016146.443101:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016146.443106:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016146.443112:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.443119:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016146.443126:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016146.443127:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d2bb 00000080:00010000:2.0:1734016146.443130:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2bb from it ffff8800a7343cd8 00010000:00010000:2.0:1734016146.443132:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.443138:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016146.443146:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.443150:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.443156:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.443157:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c30, osc lock ffff8800aafce5f0, flags 0x400001002 00000008:00010000:2.0:1734016146.443159:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.443161:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c70, osc lock ffff8800aafcf7c0, flags 0x400001002 00010000:00010000:2.0:1734016146.443165:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.443172:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.443178:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.443184:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.443192:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.443198:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.443204:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.443208:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016146.443244:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 264 00000020:00010000:2.0:1734016146.443248:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.443250:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.443252:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2c30 (ffff88012db7c7e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.443255:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2c70 (ffff8800a72d9098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.443257:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dd240 (ffff8800a72d9ab8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.443259:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2c30 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.443261:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2c70 (ffff8800a72d9ab8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.443266:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016146.443278:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:2.0:1734016146.443280:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:2.0:1734016146.443287:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016146.443295:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1758, 1760] 00000008:00010000:2.0:1734016146.443302:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016146.443304:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8730, osc lock ffff8800aafcf7c0, flags 0x0 00010000:00010000:2.0:1734016146.443311:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.443319:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3006464 3018751) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.443325:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.443331:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.443340:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016146.443348:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 264 00000100:00100000:2.0:1734016146.443425:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016146.443436:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332548480:192.168.201.152@tcp:22:fsx.0 00000100:00100000:2.0:1734016146.448901:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332548480:192.168.201.152@tcp:22:fsx.0 00000100:00100000:2.0:1734016146.455127:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016146.455212:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016146.455296:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334648192 00000100:00100000:2.0:1734016146.455312:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334648192 00000100:00100000:2.0:1734016146.455363:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334648320 00000100:00100000:2.0:1734016146.455365:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334648320 00000100:00100000:2.0:1734016146.455384:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14790 00000100:00100000:2.0:1734016146.455395:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73c8a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334648192:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.455420:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.455437:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d3be lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1867776->1871871) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31787fe expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.455477:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 187, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.455480:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 240 00000020:00010000:2.0:1734016146.455483:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.455484:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 240 00000100:00100000:2.0:1734016146.455573:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73c8a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334648192:12345-192.168.201.152@tcp:106: Request processed in 180us (464us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.455581:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14790 00000100:00100000:2.0:1734016146.455601:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14791 00000100:00100000:2.0:1734016146.455605:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73cbb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334648320:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.455614:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.455621:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.455633:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 187, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.455634:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 48 00000020:00010000:2.0:1734016146.455636:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 3, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.455637:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 48 00000100:00100000:2.0:1734016146.455654:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73cbb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334648320:12345-192.168.201.152@tcp:106: Request processed in 47us (442us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.455658:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14791 00000100:00100000:2.0:1734016146.468781:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016146.469476:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016146.473391:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 288 00000020:00010000:2.0:1734016146.473399:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dc920 (ffff8800a72d9128/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.473403:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dc920 (ffff8800a72d9128/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.473409:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40aab0 (ffff8800a72d9cf8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.473413:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa40aaf0 (ffff8800a72d9f38/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.473416:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dc920 (ffff8800a72d9128/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.473418:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40aab0 (ffff8800a72d9128/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.473424:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa40aaf0 (ffff8800a72d9128/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.473443:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.473462:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9625452 00000080:00200000:2.0:1734016146.473556:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:2.0:1734016146.473570:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:2.0:1734016146.473588:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016146.473603:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [122, 124] 00000008:00010000:2.0:1734016146.473619:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016146.473623:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8730, osc lock ffff8800aafce5f0, flags 0x0 00010000:00010000:2.0:1734016146.473657:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.473665:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [499712->511999] (req 499712->511999) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.473710:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [499712->511999] (req 499712->511999) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.473743:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332549120:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016146.473841:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2c00 going to sleep for 11 seconds 00000100:00100000:2.0:1734016146.476752:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016146.480923:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332549120:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016146.480962:0:10366:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [499712->511999] (req 499712->511999) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d3178821 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.480978:0:10366:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [499712->511999] (req 499712->511999) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178821 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.480986:0:10366:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [499712->511999] (req 499712->511999) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178821 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.486163:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000020:00010000:2.0:1734016146.500446:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a84e8 (ffff8800a72d9128/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.500452:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a84e8 (ffff8800a72d9128/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.500463:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8730 (ffff8800a72d9f38/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016146.500481:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 499712->511999) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178821 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.500500:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 499712->511999) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178821 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.500525:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a84e8 (ffff8800a72d9128/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.500528:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8730 (ffff8800a72d9128/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.500554:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.500593:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016146.500602:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016146.500639:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016146.500668:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.500681:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016146.500703:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016146.500707:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d2c2 00000080:00010000:2.0:1734016146.500714:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2c2 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016146.500719:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.500726:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016146.500756:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.500769:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.500792:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.500794:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c30, osc lock ffff8800aafce5f0, flags 0x400001002 00000008:00010000:2.0:1734016146.500801:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.500802:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2c70, osc lock ffff8800aafce720, flags 0x400001002 00010000:00010000:2.0:1734016146.500818:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 499712->511999) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178821 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.500825:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 499712->511999) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178821 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.500838:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 499712->511999) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178821 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.500846:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 499712->511999) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178821 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.500871:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d3e8 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.500877:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d3e8 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.506740:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334648960 00000100:00100000:2.0:1734016146.506758:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334648960 00000100:00100000:2.0:1734016146.506833:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14795 00000100:00100000:2.0:1734016146.506843:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334648960:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.506869:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.506883:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.506921:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.506925:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 48 00000020:00010000:2.0:1734016146.506928:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 3, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.506930:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 48 00000100:00100000:2.0:1734016146.507017:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334648960:12345-192.168.201.152@tcp:106: Request processed in 175us (506us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.507026:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14795 00010000:00010000:2.0:1734016146.517474:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d3e8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.517525:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d3e8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.517548:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d3e8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.517563:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d3e8 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.517569:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d3e8 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.517599:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016146.517634:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d3e8 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.517640:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d3e8 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.517647:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d3e8 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.517658:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbc000 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016146.517668:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d3e8 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.517692:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016146.517701:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d3e8 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.517709:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbc000/0x37cab3886ce5d3e8 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.517774:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332549504:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016146.527937:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334649088 00000100:00100000:2.0:1734016146.527960:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334649088 00000100:00100000:2.0:1734016146.528031:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14796 00000100:00100000:2.0:1734016146.528044:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334649088:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.528074:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.528094:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.528142:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.528145:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 48 00000020:00010000:2.0:1734016146.528148:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 3, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.528150:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 48 00000100:00100000:2.0:1734016146.528280:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334649088:12345-192.168.201.152@tcp:106: Request processed in 236us (917us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.528288:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14796 00000080:00200000:2.0:1734016146.538634:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 288 00000020:00010000:2.0:1734016146.538651:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dc920 (ffff8800a72d9128/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.538656:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dc920 (ffff8800a72d9128/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.538665:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86760f0 (ffff8800a72d9cf8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.538673:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676130 (ffff8800a72d9f38/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.538675:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dc920 (ffff8800a72d9128/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.538677:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86760f0 (ffff8800a72d9128/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.538685:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676130 (ffff8800a72d9128/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.538701:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.538721:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9625452 00000080:00200008:2.0:1734016146.538849:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 4758456, type: SEEK_SET 00000080:00200008:2.0:1734016146.538853:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 4758456, type: SEEK_SET, rc = 4758456 00000080:00200008:2.0:1734016146.538867:0:10366:0:(file.c:2466:do_file_write_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 4758456, count: 6444 00000080:00200000:2.0:1734016146.538877:0:10366:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: write ppos: 4758456, bytes: 6444 00000080:00200000:2.0:1734016146.538886:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:2.0:1734016146.538897:0:10366:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00200000:2.0:1734016146.538901:0:10366:0:(cl_io.c:195:cl_io_rw_init()) io range: 2 [4758456, 4764900) 0 0 00000080:00200000:2.0:1734016146.538907:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016146.538931:0:10366:0:(file.c:1893:ll_file_io_generic()) Range lock [1161, 1163] 00020000:00200000:2.0:1734016146.538945:0:10366:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 1 chunk: [4758456, 4764900) 4764900, 6444 00020000:00200000:2.0:1734016146.538950:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016146.538978:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1161, 1163] 00000008:00010000:2.0:1734016146.539005:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016146.539013:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8730, osc lock ffff8800aafcf560, flags 0x0 00010000:00010000:2.0:1734016146.539053:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.539063:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [561152->573439] (req 561152->573439) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.539121:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [561152->573439] (req 561152->573439) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.539169:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332549760:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016146.539328:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2900 going to sleep for 11 seconds 00000100:00100000:2.0:1734016146.550524:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016146.561900:0:4737:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016146.562019:0:4737:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1734016146.562087:0:4737:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 2/0,0 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.562095:0:4737:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1734016146.562123:0:4737:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.562130:0:4737:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff8800a6c21440/0x37cab3886ce5d2b4 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d31786e6 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.562153:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016146.562155:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016146.562158:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1734016146.562164:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016146.562180:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332550016:192.168.201.152@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1734016146.565629:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016146.566395:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14798 00000100:00100000:2.0:1734016146.566403:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012b52e050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6472:x1818247334649344:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016146.566424:0:4728:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [561152->573439] (req 561152->573439) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.566435:0:4728:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [561152->573439] (req 561152->573439) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.566443:0:4728:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.566457:0:4728:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.566464:0:4728:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.566473:0:4728:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.566686:0:4728:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff88012dbbf840) 00000100:00100000:2.0:1734016146.567777:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012b52e050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6472:x1818247334649344:12345-192.168.201.152@tcp:105: Request processed in 1369us (2156us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.567788:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14798 00000100:00100000:2.0:1734016146.584630:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332550016:192.168.201.152@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1734016146.688582:0:10366:0:(rw26.c:1006:ll_write_end()) pos 4763648, len 1252, copied 1252 00000080:00200000:2.0:1734016146.688596:0:10366:0:(rw26.c:1044:ll_write_end()) page@ffff8801301b9000[3 ffff8800b6e5c788 2 1 ffff8800a70dc980] 00000080:00200000:2.0:1734016146.688599:0:10366:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea00049a6480 2fffff0000082d 2:0 ffff8801301b9000 1163 lru 00000080:00200000:2.0:1734016146.688607:0:10366:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8801301b9060 139: 1< 1 - - > 2< 569344 0 4096 0x0 0x40009 | ffff8800a947c758 ffff8800a86f29c0 > 3< 0 0 > 4< 0 0 8 489881600 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1734016146.688609:0:10366:0:(rw26.c:1044:ll_write_end()) end page@ffff8801301b9000 00000080:00200000:2.0:1734016146.688610:0:10366:0:(rw26.c:1044:ll_write_end()) queued page: 3. 00000080:00200000:2.0:1734016146.688618:0:10366:0:(vvp_io.c:1218:vvp_io_write_commit()) commit async pages: 3, from 3000, to 1252 00000008:00010000:2.0:1734016146.688649:0:10366:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800aa7e2b40 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.688668:0:10366:0:(osc_cache.c:2323:osc_queue_async_io()) ### extent: ffff8800aa7e2b40 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.688681:0:10366:0:(osc_cache.c:2323:osc_queue_async_io()) ### extent: ffff8800aa7e2b40 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016146.688693:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af698248, count 3, dirtied 3 00000080:00200000:2.0:1734016146.688697:0:10366:0:(vvp_io.c:1241:vvp_io_write_commit()) Committed 3 pages 6444 bytes, tot: 6444 00000080:00200000:2.0:1734016146.688700:0:10366:0:(vvp_io.c:1405:vvp_io_write_start()) f16b.sanityn: write bytes 0, result: 6444 00000020:00010000:2.0:1734016146.688719:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a84e8 (ffff8800a72d9128/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.688722:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a84e8 (ffff8800a72d9128/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.688725:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8730 (ffff8800a72d9f38/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016146.688732:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.688742:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.688761:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a84e8 (ffff8800a72d9128/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.688764:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8730 (ffff8800a72d9128/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.688772:0:10366:0:(vvp_io.c:533:vvp_io_advance()) advancing 6444 bytes 00000080:00200000:2.0:1734016146.688781:0:10366:0:(file.c:1930:ll_file_io_generic()) Range unlock [1161, 1163] 00000080:00200000:2.0:1734016146.688821:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016146.688824:0:10366:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 2 io complete with rc: 0, result: 6444, restart: 0 00000080:00200000:2.0:1734016146.688846:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 304 00000080:00200000:2.0:1734016146.688853:0:10366:0:(file.c:2044:ll_file_io_generic()) iot: 2, result: 6444 00000080:00200008:2.0:1734016146.688891:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016146.688897:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016146.688916:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016146.688935:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.688948:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016146.688968:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016146.688972:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d2c2 00000080:00010000:2.0:1734016146.688979:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2c2 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016146.688983:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.688989:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016146.689011:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.689024:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.689047:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.689050:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2930, osc lock ffff8800aafcf560, flags 0x400001002 00000008:00010000:2.0:1734016146.689056:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.689057:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2970, osc lock ffff8800aafcfc80, flags 0x400001002 00010000:00010000:2.0:1734016146.689073:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 499712->511999) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178821 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.689082:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 499712->511999) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178821 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.689093:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 499712->511999) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178821 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.689098:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 499712->511999) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178821 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.689114:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.689120:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.689126:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.689130:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016146.689138:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 304 00000020:00010000:2.0:1734016146.689140:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.689141:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.689143:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2930 (ffff8800a72d97e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.689145:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2970 (ffff8800a72d9488/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.689146:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.689148:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2930 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.689150:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2970 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.689155:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.689170:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016146.689174:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.689177:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.689182:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.689184:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2930, osc lock ffff8800aafcfc80, flags 0x400001002 00000008:00010000:2.0:1734016146.689187:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.689188:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2970, osc lock ffff8800aafcf560, flags 0x400001002 00010000:00010000:2.0:1734016146.689193:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 499712->511999) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178821 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.689199:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 499712->511999) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178821 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.689205:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 499712->511999) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178821 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.689210:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 499712->511999) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178821 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.689248:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.689255:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.689261:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.689266:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016146.689273:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 304 00000020:00010000:2.0:1734016146.689275:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.689278:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.689280:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2930 (ffff8800a72d9488/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.689282:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2970 (ffff8800a72d97e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.689284:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.689285:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2930 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.689287:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2970 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.689291:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.689299:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9625452 00000080:00200008:2.0:1734016146.689312:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016146.689318:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016146.689329:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016146.689342:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.689350:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016146.689356:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016146.689358:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d2bb 00000080:00010000:2.0:1734016146.689361:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2bb from it ffff8800a7343cd8 00010000:00010000:2.0:1734016146.689364:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.689369:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016146.689382:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.689386:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.689393:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.689395:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2930, osc lock ffff8800aafcf560, flags 0x400001002 00000008:00010000:2.0:1734016146.689398:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.689399:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2970, osc lock ffff8800aafcfc80, flags 0x400001002 00010000:00010000:2.0:1734016146.689417:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d3fd lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.689423:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdd40/0x37cab3886ce5d3fd lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.690653:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d404 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.690662:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d404 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.690701:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332551040:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016146.694107:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016146.694203:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00010000:00010000:2.0:1734016146.698471:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d404 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.698504:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d404 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.698569:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d404 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.698576:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d404 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.698585:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d404 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.698607:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016146.698617:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d404 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.698623:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d404 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.698632:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d404 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.698642:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbcfc0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016146.698649:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d404 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.698666:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016146.698674:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d404 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.698681:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d404 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.698741:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332551040:192.168.201.152@tcp:101:fsx.0 00000080:00200000:2.0:1734016146.699642:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 304 00000020:00010000:2.0:1734016146.699651:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.699657:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.699665:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2930 (ffff8800a72d97e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.699671:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2970 (ffff8800a72d9488/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.699675:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.699677:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2930 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.699684:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2970 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.699709:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.699749:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016146.699758:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.699768:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.699792:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.699794:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2930, osc lock ffff8800aafcfc80, flags 0x400001002 00000008:00010000:2.0:1734016146.699800:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.699801:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2970, osc lock ffff8800aafcf560, flags 0x400001002 00010000:00010000:2.0:1734016146.699830:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d40b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.699839:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d40b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.699918:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016146.699932:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332551168:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016146.700022:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d412 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.700030:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d412 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.700052:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332551296:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016146.703844:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016146.703915:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00010000:00010000:2.0:1734016146.709992:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d40b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.710004:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d40b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.710013:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d40b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.710018:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d40b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.710024:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d40b lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.710037:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5431148, setting rss=5431148; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016146.710172:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d40b lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.710180:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d40b lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.710193:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d40b lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.710199:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbdf80 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016146.710204:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d40b lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.710246:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016146.710250:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d40b lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.710255:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d40b lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.710290:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332551168:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016146.716098:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016146.716524:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016146.716551:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334651008 00000100:00100000:2.0:1734016146.716554:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334651008 00000100:00100000:2.0:1734016146.716576:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14804 00000100:00100000:2.0:1734016146.716581:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73c9180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334651008:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016146.716588:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 499712->511999) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178821 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.716596:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d3e1 lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 499712->511999) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178821 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.716612:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 2049, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016146.716614:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 240 00000020:00010000:2.0:1734016146.716617:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8e10[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016146.716618:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 240 00000100:00100000:2.0:1734016146.716639:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73c9180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334651008:12345-192.168.201.152@tcp:106: Request processed in 59us (116us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.716645:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14804 00000080:00200000:2.0:1734016146.721193:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 304 00000020:00010000:2.0:1734016146.721201:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.721204:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.721211:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a708c1b0 (ffff8800a72d97e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.721346:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a708c1f0 (ffff8800a72d9488/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.721348:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.721350:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a708c1b0 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.721355:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a708c1f0 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.721368:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.721404:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016146.721412:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.721422:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.721445:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.721448:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f30, osc lock ffff8800aafce980, flags 0x400001002 00000008:00010000:2.0:1734016146.721454:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.721455:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f70, osc lock ffff8800aafcfdb0, flags 0x400001002 00010000:00010000:2.0:1734016146.721486:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d427 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.721500:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d427 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.721578:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d42e lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.721586:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d42e lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.725175:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016146.725556:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016146.739204:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 304 00000020:00010000:2.0:1734016146.739237:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.739243:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.739249:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f30 (ffff8800a72d9488/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.739256:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f70 (ffff8800a72d97e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.739260:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.739262:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f30 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.739275:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f70 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.739305:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.739321:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9625452 00000080:00200008:2.0:1734016146.739432:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 186544, type: SEEK_SET 00000080:00200008:2.0:1734016146.739439:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 186544, type: SEEK_SET, rc = 186544 00000080:00200008:2.0:1734016146.739452:0:10366:0:(file.c:2466:do_file_write_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 186544, count: 5199 00000080:00200000:2.0:1734016146.739463:0:10366:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: write ppos: 186544, bytes: 5199 00000080:00200000:2.0:1734016146.739471:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:2.0:1734016146.739483:0:10366:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b62c6a28[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00200000:2.0:1734016146.739486:0:10366:0:(cl_io.c:195:cl_io_rw_init()) io range: 2 [186544, 191743) 0 0 00000080:00200000:2.0:1734016146.739493:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016146.739526:0:10366:0:(file.c:1893:ll_file_io_generic()) Range lock [45, 46] 00020000:00200000:2.0:1734016146.739537:0:10366:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 0 chunk: [186544, 191743) 191743, 5199 00020000:00200000:2.0:1734016146.739543:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016146.739567:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [45, 46] 00000008:00010000:2.0:1734016146.739591:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016146.739598:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8730, osc lock ffff8800aafcfdb0, flags 0x0 00010000:00010000:2.0:1734016146.739648:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.739662:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [184320->192511] (req 184320->192511) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.739726:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [184320->192511] (req 184320->192511) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.739764:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8380 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332551936:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016146.739883:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2240 going to sleep for 11 seconds 00000100:00100000:2.0:1734016146.754532:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016146.780942:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016146.793734:0:10366:0:(rw26.c:1006:ll_write_end()) pos 186544, len 1872, copied 1872 00000080:00200000:2.0:1734016146.793745:0:10366:0:(rw26.c:1044:ll_write_end()) page@ffff8800a703cbd0[3 ffff8800b62c6aa8 2 1 ffff8800a70df290] 00000080:00200000:2.0:1734016146.793748:0:10366:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002a34400 1fffff0000080d 3:0 ffff8800a703cbd0 45 lru 00000080:00200000:2.0:1734016146.793756:0:10366:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800a703cc30 45: 1< 1 - - > 2< 184320 0 4096 0x0 0x40009 | ffff8800a97f0708 ffff88012b46a9c0 > 3< 0 0 > 4< 0 0 8 489881600 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1734016146.793758:0:10366:0:(rw26.c:1044:ll_write_end()) end page@ffff8800a703cbd0 00000080:00200000:2.0:1734016146.793759:0:10366:0:(rw26.c:1044:ll_write_end()) queued page: 1. 00000080:00200000:2.0:1734016146.793763:0:10366:0:(rw26.c:819:ll_write_begin()) Writing 46 of 0 to 3327 bytes 00000100:00100000:2.0:1734016146.793904:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:2.0:1734016146.793916:0:1970:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e2cc0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.793961:0:1970:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (188416 192511) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016146.794000:0:1970:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73c8000 - object 0x280000400:7 offset 188416<>192512 00000100:00100000:2.0:1734016146.794022:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332552576:192.168.201.152@tcp:3:fsx.0 00000080:00200000:2.0:1734016146.800273:0:10366:0:(rw26.c:1006:ll_write_end()) pos 188416, len 3327, copied 3327 00000080:00200000:2.0:1734016146.800285:0:10366:0:(rw26.c:1044:ll_write_end()) page@ffff88012bfa4ca8[3 ffff8800b62c6aa8 2 1 ffff8800a70df290] 00000080:00200000:2.0:1734016146.800288:0:10366:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002b86640 1fffff0000080d 3:0 ffff88012bfa4ca8 46 lru 00000080:00200000:2.0:1734016146.800297:0:10366:0:(rw26.c:1044:ll_write_end()) osc-page@ffff88012bfa4d08 46: 1< 1 - - > 2< 188416 0 4096 0x0 0x40009 | ffff8800a97f0708 ffff88012b46a9c0 > 3< 0 0 > 4< 0 0 8 489881600 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1734016146.800298:0:10366:0:(rw26.c:1044:ll_write_end()) end page@ffff88012bfa4ca8 00000080:00200000:2.0:1734016146.800299:0:10366:0:(rw26.c:1044:ll_write_end()) queued page: 2. 00000080:00200000:2.0:1734016146.800310:0:10366:0:(vvp_io.c:1218:vvp_io_write_commit()) commit async pages: 2, from 2224, to 3327 00000008:00010000:2.0:1734016146.800346:0:10366:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800aa7e2840 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.800370:0:10366:0:(osc_cache.c:2323:osc_queue_async_io()) ### extent: ffff8800aa7e2840 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016146.800384:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 2, dirtied 2 00000080:00200000:2.0:1734016146.800388:0:10366:0:(vvp_io.c:1241:vvp_io_write_commit()) Committed 2 pages 5199 bytes, tot: 5199 00000080:00200000:2.0:1734016146.800392:0:10366:0:(vvp_io.c:1405:vvp_io_write_start()) f16b.sanityn: write bytes 0, result: 5199 00000020:00010000:2.0:1734016146.800414:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a85d8 (ffff8800a72d9f38/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.800418:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a85d8 (ffff8800a72d9f38/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.800423:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8730 (ffff8800a72d97e8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016146.800433:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.800450:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.800472:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a85d8 (ffff8800a72d9f38/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.800473:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8730 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.800485:0:10366:0:(vvp_io.c:533:vvp_io_advance()) advancing 5199 bytes 00000080:00200000:2.0:1734016146.800497:0:10366:0:(file.c:1930:ll_file_io_generic()) Range unlock [45, 46] 00000080:00200000:2.0:1734016146.800541:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016146.800545:0:10366:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 2 io complete with rc: 0, result: 5199, restart: 0 00000080:00200000:2.0:1734016146.800567:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 304 00000080:00200000:2.0:1734016146.800577:0:10366:0:(file.c:2044:ll_file_io_generic()) iot: 2, result: 5199 00000080:00200008:2.0:1734016146.800613:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016146.800624:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016146.801168:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016146.801189:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.801203:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016146.801246:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016146.801250:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d2c2 00000080:00010000:2.0:1734016146.801257:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2c2 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016146.801260:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.801265:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016146.801292:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.801311:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.801331:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.801335:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e23f0, osc lock ffff8800aafcfdb0, flags 0x400001002 00000008:00010000:2.0:1734016146.801343:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.801345:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2430, osc lock ffff8800aafce980, flags 0x400001002 00010000:00010000:2.0:1734016146.801377:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d43c lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.801384:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d43c lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.801475:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.801484:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.801497:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.801505:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.801554:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016146.801579:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332552704:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016146.808760:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016146.817383:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 304 00000020:00010000:2.0:1734016146.817393:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.817398:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.817405:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e23f0 (ffff8800a72d97e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.817408:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2430 (ffff8800a72d9488/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.817411:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.817413:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e23f0 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.817419:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2430 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.817435:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.817457:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016146.817463:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.817469:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.817483:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.817486:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e23f0, osc lock ffff8800aafce980, flags 0x400001002 00000008:00010000:2.0:1734016146.817492:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.817494:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2430, osc lock ffff8800aafcfdb0, flags 0x400001002 00010000:00010000:2.0:1734016146.817527:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d443 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.817534:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d443 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.817585:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.817592:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.817600:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.817607:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.823356:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016146.833791:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 304 00000020:00010000:2.0:1734016146.833812:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.833818:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.833827:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e23f0 (ffff8800a72d9488/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.833835:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2430 (ffff8800a72d97e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.833840:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.833842:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e23f0 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.833851:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2430 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.833876:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.833887:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9625452 00000080:00200008:2.0:1734016146.833911:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016146.833919:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016146.833940:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016146.833961:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.833974:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016146.833985:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016146.833987:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d2c2 00000080:00010000:2.0:1734016146.833995:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2c2 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016146.833998:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.834005:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016146.834027:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.834035:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.834053:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.834057:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e23f0, osc lock ffff8800aafcfdb0, flags 0x400001002 00000008:00010000:2.0:1734016146.834060:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.834061:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2430, osc lock ffff8800aafce980, flags 0x400001002 00010000:00010000:2.0:1734016146.834084:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d44a lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.834090:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d44a lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.834177:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.834184:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.834193:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.834199:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.838275:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016146.848120:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 304 00000020:00010000:2.0:1734016146.848129:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.848133:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.848141:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e23f0 (ffff8800a72d97e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.848146:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2430 (ffff8800a72d9488/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.848150:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.848152:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e23f0 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.848156:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2430 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.848175:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.848201:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016146.848208:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.848243:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.848262:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.848265:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e23f0, osc lock ffff8800aafce980, flags 0x400001002 00000008:00010000:2.0:1734016146.848271:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.848273:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2430, osc lock ffff8800aafcfdb0, flags 0x400001002 00010000:00010000:2.0:1734016146.848305:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d451 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.848314:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d451 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.848402:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.848415:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.848427:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.848435:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.858698:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016146.874008:0:10366:0:(rw26.c:1006:ll_write_end()) pos 88032, len 2080, copied 2080 00000080:00200000:2.0:1734016146.874022:0:10366:0:(rw26.c:1044:ll_write_end()) page@ffff8800a703cca8[3 ffff8800b62c6aa8 2 1 ffff8800a70df290] 00000080:00200000:2.0:1734016146.874026:0:10366:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002a34440 1fffff0000080d 3:0 ffff8800a703cca8 21 lru 00000080:00200000:2.0:1734016146.874035:0:10366:0:(rw26.c:1044:ll_write_end()) osc-page@ffff8800a703cd08 21: 1< 1 - - > 2< 86016 0 4096 0x0 0x40009 | ffff8800a97f0708 ffff88012b46a9c0 > 3< 0 0 > 4< 0 0 8 489848832 - | - - + - > 5< - - + - | 0 - | 2 - -> 00000080:00200000:2.0:1734016146.874037:0:10366:0:(rw26.c:1044:ll_write_end()) end page@ffff8800a703cca8 00000080:00200000:2.0:1734016146.874038:0:10366:0:(rw26.c:1044:ll_write_end()) queued page: 1. 00000080:00200000:2.0:1734016146.874044:0:10366:0:(rw26.c:819:ll_write_begin()) Writing 22 of 0 to 4096 bytes 00000080:00200000:2.0:1734016146.874099:0:10366:0:(rw26.c:1006:ll_write_end()) pos 90112, len 4096, copied 4096 00000080:00200000:2.0:1734016146.874103:0:10366:0:(rw26.c:1044:ll_write_end()) page@ffff88012bfa4af8[3 ffff8800b62c6aa8 2 1 ffff8800a70df290] 00000080:00200000:2.0:1734016146.874105:0:10366:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002a07f40 1fffff00000805 3:0 ffff88012bfa4af8 22 lru 00000080:00200000:2.0:1734016146.874109:0:10366:0:(rw26.c:1044:ll_write_end()) osc-page@ffff88012bfa4b58 22: 1< 0 - - > 2< 90112 0 0 0x0 0x0 | ffff8800a97f0708 ffff88012b46a9c0 > 3< 0 0 > 4< 0 0 8 489848832 - | - - + - > 5< - - + - | 0 - | 2 - -> 00000080:00200000:2.0:1734016146.874110:0:10366:0:(rw26.c:1044:ll_write_end()) end page@ffff88012bfa4af8 00000080:00200000:2.0:1734016146.874111:0:10366:0:(rw26.c:1044:ll_write_end()) queued page: 2. 00000080:00200000:2.0:1734016146.874112:0:10366:0:(rw26.c:819:ll_write_begin()) Writing 23 of 0 to 602 bytes 00000100:00100000:2.0:1734016146.874198:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:2.0:1734016146.874208:0:1970:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e2300 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.874270:0:1970:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (94208 98303) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016146.874318:0:1970:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73c9180 - object 0x280000400:7 offset 94208<>98304 00000100:00100000:2.0:1734016146.874337:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332553600:192.168.201.152@tcp:3:fsx.0 00000080:00200000:2.0:1734016146.884898:0:10366:0:(rw26.c:1006:ll_write_end()) pos 94208, len 602, copied 602 00000080:00200000:2.0:1734016146.884912:0:10366:0:(rw26.c:1044:ll_write_end()) page@ffff88012bfa4a20[3 ffff8800b62c6aa8 2 1 ffff8800a70df290] 00000080:00200000:2.0:1734016146.884915:0:10366:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002a73b00 1fffff0000080d 3:0 ffff88012bfa4a20 23 lru 00000080:00200000:2.0:1734016146.884924:0:10366:0:(rw26.c:1044:ll_write_end()) osc-page@ffff88012bfa4a80 23: 1< 1 - - > 2< 94208 0 4096 0x0 0x40009 | ffff8800a97f0708 ffff88012b46a9c0 > 3< 0 0 > 4< 0 0 8 489848832 - | - - + - > 5< - - + - | 0 - | 2 - -> 00000080:00200000:2.0:1734016146.884926:0:10366:0:(rw26.c:1044:ll_write_end()) end page@ffff88012bfa4a20 00000080:00200000:2.0:1734016146.884928:0:10366:0:(rw26.c:1044:ll_write_end()) queued page: 3. 00000080:00200000:2.0:1734016146.884937:0:10366:0:(vvp_io.c:1218:vvp_io_write_commit()) commit async pages: 3, from 2016, to 602 00000008:00010000:2.0:1734016146.884974:0:10366:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800aa7e2900 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.885000:0:10366:0:(osc_cache.c:2323:osc_queue_async_io()) ### extent: ffff8800aa7e2900 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016146.885008:0:10366:0:(osc_cache.c:2323:osc_queue_async_io()) ### extent: ffff8800aa7e2900 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016146.885018:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 3, dirtied 3 00000080:00200000:2.0:1734016146.885023:0:10366:0:(vvp_io.c:1241:vvp_io_write_commit()) Committed 3 pages 6778 bytes, tot: 6778 00000080:00200000:2.0:1734016146.885025:0:10366:0:(vvp_io.c:1405:vvp_io_write_start()) f16b.sanityn: write bytes 0, result: 6778 00000020:00010000:2.0:1734016146.885045:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a85d8 (ffff8800a72d9f38/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.885048:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a85d8 (ffff8800a72d9f38/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.885053:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f8fb0 (ffff8800a72d97e8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016146.885064:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.885082:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016146.885103:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a85d8 (ffff8800a72d9f38/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.885105:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f8fb0 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.885119:0:10366:0:(vvp_io.c:533:vvp_io_advance()) advancing 6778 bytes 00000080:00200000:2.0:1734016146.885132:0:10366:0:(file.c:1930:ll_file_io_generic()) Range unlock [21, 23] 00000080:00200000:2.0:1734016146.885142:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016146.885146:0:10366:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 2 io complete with rc: 0, result: 6778, restart: 0 00000080:00200000:2.0:1734016146.885168:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 304 00000080:00200000:2.0:1734016146.885178:0:10366:0:(file.c:2044:ll_file_io_generic()) iot: 2, result: 6778 00000080:00200008:2.0:1734016146.885214:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016146.885250:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016146.885278:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016146.885301:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.885313:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016146.885335:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016146.885339:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d2c2 00000080:00010000:2.0:1734016146.885345:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2c2 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016146.885347:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016146.885353:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016146.885376:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.885393:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.885418:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.885420:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e29f0, osc lock ffff8800aafcee40, flags 0x400001002 00000008:00010000:2.0:1734016146.885428:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.885430:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2a30, osc lock ffff8800aafced10, flags 0x400001002 00010000:00010000:2.0:1734016146.885457:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d458 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.885465:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d458 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.885564:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.885573:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.885584:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.885591:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.888560:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016146.896791:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 304 00000020:00010000:2.0:1734016146.896800:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.896804:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.896810:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e29f0 (ffff8800a72d97e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.896815:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2a30 (ffff8800a72d9cf8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.896819:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.896821:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e29f0 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.896826:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2a30 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.896844:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.896873:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016146.896881:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.896892:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.896911:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.896914:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e29f0, osc lock ffff8800aafced10, flags 0x400001002 00000008:00010000:2.0:1734016146.896919:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.896921:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2a30, osc lock ffff8800aafcee40, flags 0x400001002 00010000:00010000:2.0:1734016146.896945:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d45f lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.896952:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d45f lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.897026:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.897033:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.897042:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.897053:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 5/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.901672:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016146.906483:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 304 00000020:00010000:2.0:1734016146.906494:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.906513:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.906518:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e29f0 (ffff8800a72d9cf8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.906523:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2a30 (ffff8800a72d97e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.906527:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.906529:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e29f0 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.906536:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2a30 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.906559:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016146.906577:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9625452 00000080:00a00000:2.0:1734016146.906710:0:10366:0:(llite_mmap.c:611:ll_file_mmap()) VFS_Op: fid=[0x200000401:0xc:0x0] vma=ffff8800a91820d8 start=0x7fef68fb7000 end=0x7fef68fb9000 vm_flags=0xfb 00000080:00200000:2.0:1734016146.906732:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016146.906745:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016146.906766:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.906770:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e29f0, osc lock ffff8800aafcee40, flags 0x400001002 00000008:00010000:2.0:1734016146.906777:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016146.906778:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2a30, osc lock ffff8800aafced10, flags 0x400001002 00010000:00010000:2.0:1734016146.906802:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.906821:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.906839:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 7/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.906847:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 6/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.906889:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d466 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.906896:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbf600/0x37cab3886ce5d466 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.906981:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332553984:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016146.915099:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016146.920868:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 304 00000020:00010000:2.0:1734016146.920879:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_release():233 00000020:00010000:2.0:1734016146.920883:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.920887:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e29f0 (ffff8800a72d97e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.920891:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2a30 (ffff8800a72d9cf8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016146.920893:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.920894:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e29f0 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016146.920902:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2a30 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016146.920923:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016146.920997:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 1621 00000080:00200000:2.0:1734016146.921008:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016146.921025:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016146.921039:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1621, 1621] 00000008:00010000:2.0:1734016146.921058:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016146.921063:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8730, osc lock ffff8800aafced10, flags 0x0 00010000:00010000:2.0:1734016146.921093:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d46d lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.921103:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d46d lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [2445312->2449407] (req 2445312->2449407) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.921140:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d46d lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [2445312->2449407] (req 2445312->2449407) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016146.921169:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8700 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332554112:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016146.921276:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e20c0 going to sleep for 11 seconds 00000100:00100000:2.0:1734016146.927320:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016146.933699:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8700 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332554112:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016146.933750:0:10366:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d46d lrc: 4/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [2445312->2449407] (req 2445312->2449407) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31788ad expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.933765:0:10366:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d46d lrc: 4/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [2445312->2449407] (req 2445312->2449407) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31788ad expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.933775:0:10366:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d46d lrc: 4/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [2445312->2449407] (req 2445312->2449407) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31788ad expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016146.934466:0:5236:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016146.939437:0:5236:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1734016146.939576:0:5236:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 2/0,0 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.939592:0:5236:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1734016146.940128:0:5236:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.940138:0:5236:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d3f6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 561152->573439) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d3178836 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.940170:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016146.940172:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016146.940179:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1734016146.948133:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334652672 00000100:00100000:2.0:1734016146.948157:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334652672 00000100:00100000:2.0:1734016146.948416:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14816 00000100:00100000:2.0:1734016146.948428:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6472:x1818247334652672:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016146.948460:0:4729:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d46d lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [2445312->2449407] (req 2445312->2449407) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31788ad expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.948478:0:4729:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d46d lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [2445312->2449407] (req 2445312->2449407) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31788ad expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.948490:0:4729:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d46d lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2445312->2449407) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31788ad expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.948523:0:4729:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d46d lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2445312->2449407) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31788ad expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.948542:0:4729:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d46d lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2445312->2449407) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31788ad expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.948559:0:4729:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d46d lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2445312->2449407) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d31788ad expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016146.948607:0:4729:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff88012dbbcfc0) 00000100:00100000:2.0:1734016146.948699:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6472:x1818247334652672:12345-192.168.201.152@tcp:105: Request processed in 273us (744us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016146.948707:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14816 00000100:00100000:2.0:1734016146.956659:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332554368:192.168.201.152@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1734016146.974009:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800b341fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332555008:192.168.201.152@tcp:4:fsx.0 00000100:00100000:2.0:1734016147.001348:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016147.008204:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016147.008273:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334652928 00000100:00100000:2.0:1734016147.008279:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334652928 00000100:00100000:2.0:1734016147.008311:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14818 00000100:00100000:2.0:1734016147.008317:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73c8000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334652928:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016147.008330:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d46d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2445312->2449407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31788ad expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.008340:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d46d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2445312->2449407) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31788ad expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016147.008365:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 8, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016147.008368:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 112 00000020:00010000:2.0:1734016147.008371:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016147.008372:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 112 00000100:00100000:2.0:1734016147.008399:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73c8000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334652928:12345-192.168.201.152@tcp:106: Request processed in 82us (200us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016147.008404:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14818 00000080:00200000:2.0:1734016147.019467:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 352 00000020:00010000:2.0:1734016147.019483:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_release():233 00000020:00010000:2.0:1734016147.019487:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.019496:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676270 (ffff8800a72d9cf8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.019502:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86762b0 (ffff8800a72d9098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.019516:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70df230 (ffff8800a72d9f38/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.019520:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676270 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.019530:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86762b0 (ffff8800a72d9f38/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016147.019616:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016147.019690:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016147.019703:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016147.019719:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016147.019750:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016147.019754:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e26f0, osc lock ffff8800aafcee40, flags 0x400001002 00000008:00010000:2.0:1734016147.019767:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016147.019768:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2730, osc lock ffff8800aafcf690, flags 0x400001002 00010000:00010000:2.0:1734016147.019801:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5d482 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.019812:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5d482 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.019951:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332555520:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016147.020087:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d489 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.020093:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d489 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.020119:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016147.020125:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332555648:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016147.029004:0:4728:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016147.029025:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334653056 00000100:00100000:2.0:1734016147.029030:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334653056 00000100:00100000:2.0:1734016147.029086:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14819 00000100:00100000:2.0:1734016147.029094:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012b52d050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334653056:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016147.029108:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.029120:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016147.029141:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 8, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016147.029142:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 240 00000020:00010000:2.0:1734016147.029146:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 4, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016147.029147:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 240 00000100:00100000:2.0:1734016147.029172:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012b52d050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:10116:x1818247334653056:12345-192.168.201.152@tcp:106: Request processed in 80us (1891us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016147.029179:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14819 00010000:00010000:2.0:1734016147.034356:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d489 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.034386:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d489 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.034400:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d489 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.034408:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d489 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.034413:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d489 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016147.034436:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016147.034449:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d489 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.034453:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d489 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.034462:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d489 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.034470:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbcd80 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016147.034478:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d489 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.034496:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016147.034516:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d489 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.034522:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d489 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.034595:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332555648:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016147.034627:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5d482 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.034636:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5d482 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.034645:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5d482 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.034650:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5d482 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.034655:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5d482 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016147.034662:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5431148, setting rss=5431148; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016147.034693:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5d482 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.034699:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5d482 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.034705:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5d482 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.034709:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbdf80 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016147.034713:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5d482 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.034721:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016147.034724:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5d482 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.034730:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdf80/0x37cab3886ce5d482 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.034745:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332555520:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016147.047288:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d8c0/0x37cab3886ce5d490 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.047310:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d8c0/0x37cab3886ce5d490 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.047328:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d8c0/0x37cab3886ce5d490 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.047334:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d8c0/0x37cab3886ce5d490 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.047341:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d8c0/0x37cab3886ce5d490 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016147.047360:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5431148, setting rss=5431148; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016147.047369:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d8c0/0x37cab3886ce5d490 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.047375:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d8c0/0x37cab3886ce5d490 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.047382:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d8c0/0x37cab3886ce5d490 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.047389:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a964d8c0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016147.047393:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d8c0/0x37cab3886ce5d490 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.047405:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016147.047410:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d8c0/0x37cab3886ce5d490 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.047422:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d8c0/0x37cab3886ce5d490 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.047465:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332555776:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016147.047486:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964dd40/0x37cab3886ce5d497 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.047493:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964dd40/0x37cab3886ce5d497 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.047499:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964dd40/0x37cab3886ce5d497 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.047516:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964dd40/0x37cab3886ce5d497 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.047521:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964dd40/0x37cab3886ce5d497 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016147.047527:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016147.047543:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964dd40/0x37cab3886ce5d497 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.047548:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964dd40/0x37cab3886ce5d497 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.047558:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964dd40/0x37cab3886ce5d497 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.047563:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a964dd40 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016147.047565:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964dd40/0x37cab3886ce5d497 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.047571:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016147.047573:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964dd40/0x37cab3886ce5d497 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.047578:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964dd40/0x37cab3886ce5d497 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.047590:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332555904:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016147.054109:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334653568 00000100:00100000:2.0:1734016147.054125:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334653568 00000100:00100000:2.0:1734016147.054179:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14822 00000100:00100000:2.0:1734016147.054187:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334653568:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016147.054203:0:4729:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d46d lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2445312->2449407) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d31788ad expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.054707:0:4729:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016147.054717:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334653568:12345-192.168.201.152@tcp:104: Request processed in 529us (752us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016147.054728:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14822 00000100:00100000:2.0:1734016147.063750:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334653696 00000100:00100000:2.0:1734016147.063764:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334653696 00000100:00100000:2.0:1734016147.063816:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14823 00000100:00100000:2.0:1734016147.063824:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ca80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6472:x1818247334653696:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016147.063844:0:4729:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [2441216->2445311] (req 2441216->2445311) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.063855:0:4729:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [2441216->2445311] (req 2441216->2445311) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.063864:0:4729:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 5/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2441216->2445311) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.063876:0:4729:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2441216->2445311) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.063885:0:4729:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2441216->2445311) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.063897:0:4729:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2441216->2445311) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.063935:0:4729:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a964d440) 00000100:00100000:2.0:1734016147.063981:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ca80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6472:x1818247334653696:12345-192.168.201.152@tcp:105: Request processed in 158us (676us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016147.063986:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14823 00000020:00010000:2.0:1734016147.067398:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8d58 (ffff8800a72d9cf8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016147.067402:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8d58 (ffff8800a72d9cf8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.067405:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b35baf30 (ffff8800a72d9488/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016147.067415:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2441216->2445311) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.067426:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2441216->2445311) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016147.067442:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8d58 (ffff8800a72d9cf8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.067445:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b35baf30 (ffff8800a72d9cf8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016147.067469:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016147.067494:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016147.067518:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016147.067530:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1620, 1620] 00000008:00010000:2.0:1734016147.067543:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016147.067548:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8430, osc lock ffff8800aafcfa20, flags 0x0 00010000:00010000:2.0:1734016147.067566:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 2441216->2445311) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.067578:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2441216 2445311) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2441216->2445311) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.067588:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2441216->2445311) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.067594:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2441216->2445311) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.067603:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2441216->2445311) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016147.068788:0:10366:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800aa7e20c0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2441216->2445311) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016147.068821:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af698248, count 1, dirtied 1 00000020:00010000:2.0:1734016147.068847:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8d58 (ffff8800a72d9cf8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016147.068851:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8d58 (ffff8800a72d9cf8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.068854:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8430 (ffff8800a72d9998/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016147.068861:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2441216->2445311) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.068869:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2441216->2445311) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016147.068886:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8d58 (ffff8800a72d9cf8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.068888:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8430 (ffff8800a72d9cf8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016147.068908:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016147.068957:0:10366:0:(file.c:5185:ll_fsync()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8), start 6635520, end 6639615, datasync 1 00000080:00200000:2.0:1734016147.068972:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016147.068980:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1734016147.069002:0:10366:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e20c0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2441216->2445311) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016147.069017:0:10366:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800aa7e20c0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2441216->2445311) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.072066:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016147.072096:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800b341dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332556544:192.168.201.152@tcp:4:fsx.0 00000008:00010000:2.0:1734016147.078940:0:1971:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e20c0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2441216->2445311) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016147.078997:0:1971:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e20c0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2441216->2445311) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.079023:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332556544:192.168.201.152@tcp:4:fsx.0 00000080:00200000:2.0:1734016147.103534:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 360 00000080:00200000:2.0:1734016147.103557:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016147.104043:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016147.104053:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016147.104630:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016147.104661:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016147.104677:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016147.104700:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016147.104703:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d2c2 00000080:00010000:2.0:1734016147.104712:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2c2 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016147.104715:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016147.104722:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016147.104754:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016147.104767:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016147.104794:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016147.104797:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2630, osc lock ffff8800aafcfa20, flags 0x400001002 00000008:00010000:2.0:1734016147.104804:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016147.104805:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2670, osc lock ffff8800aafcf8f0, flags 0x400001002 00010000:00010000:2.0:1734016147.104836:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d4a5 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.104844:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d4a5 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.107461:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d49e lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2441216->2445311) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31788de expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.107496:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016147.107537:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332556928:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016147.126836:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334653824 00000100:00100000:2.0:1734016147.126849:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334653824 00000100:00100000:2.0:1734016147.126894:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14824 00000100:00100000:2.0:1734016147.126902:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334653824:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016147.126915:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.126933:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016147.126971:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 6, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016147.126974:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 240 00000020:00010000:2.0:1734016147.126978:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 4, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016147.126979:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016146 atime: 1734016146 ctime: 1734016146 blocks: 240 00000100:00100000:2.0:1734016147.127046:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334653824:12345-192.168.201.152@tcp:106: Request processed in 144us (340us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016147.127053:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14824 00010000:00010000:2.0:1734016147.137880:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d4a5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.137920:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d4a5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.137940:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d4a5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.137950:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d4a5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.137957:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d4a5 lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016147.137983:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5431148, setting rss=5431148; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016147.138062:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d4a5 lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.138070:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d4a5 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.138079:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d4a5 lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.138088:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbdd40 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016147.138094:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d4a5 lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.138113:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016147.138125:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d4a5 lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.138132:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d4a5 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.138203:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332556928:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016147.149764:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964ef40/0x37cab3886ce5d4ac lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.149793:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964ef40/0x37cab3886ce5d4ac lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.149816:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964ef40/0x37cab3886ce5d4ac lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.149822:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964ef40/0x37cab3886ce5d4ac lrc: 6/2,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.149830:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964ef40/0x37cab3886ce5d4ac lrc: 5/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016147.149849:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 5431148, setting rss=5431148; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016147.149885:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964ef40/0x37cab3886ce5d4ac lrc: 4/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.149890:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964ef40/0x37cab3886ce5d4ac lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.149897:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964ef40/0x37cab3886ce5d4ac lrc: 4/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.149904:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff8800a964ef40 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016147.149910:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964ef40/0x37cab3886ce5d4ac lrc: 5/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.149921:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016147.149927:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964ef40/0x37cab3886ce5d4ac lrc: 3/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.149937:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964ef40/0x37cab3886ce5d4ac lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.149987:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332557056:192.168.201.152@tcp:101:fsx.0 00000008:00010000:2.0:1734016147.176017:0:1971:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a708c780 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 6/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.176121:0:1971:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (2551808 2555903) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 7/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016147.176370:0:1971:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73c9880 - object 0x280000400:7 offset 0<>4194304 00000100:00100000:2.0:1734016147.305688:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016147.317758:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016147.317817:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334654208 00000100:00100000:2.0:1734016147.317832:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334654208 00000100:00100000:2.0:1734016147.317899:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14827 00000100:00100000:2.0:1734016147.317907:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73c8700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334654208:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016147.317929:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.317944:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016147.317985:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016147.317988:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016146 atime: 1734016147 ctime: 1734016146 blocks: 240 00000020:00010000:2.0:1734016147.317992:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 4, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016147.317993:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016146 atime: 1734016147 ctime: 1734016146 blocks: 240 00000100:00100000:2.0:1734016147.318057:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73c8700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334654208:12345-192.168.201.152@tcp:106: Request processed in 149us (324us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016147.318064:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14827 00000080:00200000:2.0:1734016147.326110:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 360 00000020:00010000:2.0:1734016147.326118:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70de238 (ffff8800a72d92d8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016147.326120:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70de238 (ffff8800a72d92d8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.326124:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676cf0 (ffff88012db7c6c8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.326129:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676d30 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.326132:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70de238 (ffff8800a72d92d8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.326134:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676cf0 (ffff8800a72d92d8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.326140:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676d30 (ffff8800a72d92d8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016147.326156:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016147.326171:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9625452 00000080:00200008:2.0:1734016147.326304:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 6421359, type: SEEK_SET 00000080:00200008:2.0:1734016147.326313:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 6421359, type: SEEK_SET, rc = 6421359 00000080:00200008:2.0:1734016147.326322:0:10366:0:(file.c:2466:do_file_write_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 6421359, count: 1744 00000080:00200000:2.0:1734016147.326328:0:10366:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: write ppos: 6421359, bytes: 1744 00000080:00200000:2.0:1734016147.326333:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:2.0:1734016147.326339:0:10366:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b62c6a28[0x4, 1025, [0x200000401:0xc:0x0] hash] 00000020:00200000:2.0:1734016147.326341:0:10366:0:(cl_io.c:195:cl_io_rw_init()) io range: 2 [6421359, 6423103) 0 0 00000080:00200000:2.0:1734016147.326345:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016147.326359:0:10366:0:(file.c:1893:ll_file_io_generic()) Range lock [1567, 1568] 00020000:00200000:2.0:1734016147.326367:0:10366:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 1 chunk: [6421359, 6423103) 6423103, 1744 00020000:00200000:2.0:1734016147.326371:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016147.326394:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1567, 1568] 00000008:00010000:2.0:1734016147.326410:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016147.326414:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8430, osc lock ffff8800aafcfb50, flags 0x0 00010000:00010000:2.0:1734016147.326450:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d4c1 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.326460:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d4c1 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [2224128->2232319] (req 2224128->2232319) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.326519:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d4c1 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [2224128->2232319] (req 2224128->2232319) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.326546:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9180 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332557952:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016147.326613:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2d80 going to sleep for 11 seconds 00000100:00100000:2.0:1734016147.347207:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016147.358425:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016147.468479:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334654592 00000100:00100000:2.0:1734016147.468509:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334654592 00000100:00100000:2.0:1734016147.469652:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14830 00000100:00100000:2.0:1734016147.469666:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334654592:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016147.469683:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.469694:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016147.469728:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 2049, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016147.469731:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 240 00000020:00010000:2.0:1734016147.469733:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 4, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016147.469734:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 240 00000100:00100000:2.0:1734016147.469803:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334654592:12345-192.168.201.152@tcp:106: Request processed in 139us (1509us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016147.469810:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14830 00000100:00100000:2.0:1734016147.470427:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334654720 00000100:00100000:2.0:1734016147.470436:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334654720 00000100:00100000:2.0:1734016147.470466:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14831 00000100:00100000:2.0:1734016147.470471:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341d880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334654720:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016147.470482:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d4c1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2224128->2232319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178901 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.470490:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d4c1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2224128->2232319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178901 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016147.470516:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 2049, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016147.470518:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 120 00000020:00010000:2.0:1734016147.470522:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 3, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016147.470523:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 120 00000100:00100000:2.0:1734016147.470550:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341d880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334654720:12345-192.168.201.152@tcp:106: Request processed in 78us (251us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016147.470557:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14831 00000100:00100000:2.0:1734016147.486415:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334655104 00000100:00100000:2.0:1734016147.486426:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334655104 00000100:00100000:2.0:1734016147.486478:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14835 00000100:00100000:2.0:1734016147.486482:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334655104:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016147.486506:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d4c1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2224128->2232319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178901 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.486517:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d4c1 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2224128->2232319) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178901 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016147.486543:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 2049, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016147.486546:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 120 00000020:00010000:2.0:1734016147.486549:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 3, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016147.486550:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 120 00000100:00100000:2.0:1734016147.486586:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334655104:12345-192.168.201.152@tcp:106: Request processed in 102us (240us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016147.486592:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14835 00000100:00100000:2.0:1734016147.493306:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016147.502288:0:4737:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016147.507090:0:4737:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1734016147.507343:0:4737:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d4c1 lrc: 2/0,0 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2224128->2232319) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d3178901 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.507359:0:4737:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1734016147.507891:0:4737:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d4c1 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2224128->2232319) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d3178901 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.507900:0:4737:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d4c1 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2224128->2232319) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d3178901 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.507936:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016147.507938:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016147.507945:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1734016147.514925:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016147.523323:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332559488:192.168.201.152@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1734016147.528796:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334655616 00000100:00100000:2.0:1734016147.528813:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334655616 00000100:00100000:2.0:1734016147.528886:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14838 00000100:00100000:2.0:1734016147.528898:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334655616:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016147.528924:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d4f2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2260992->2273279) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178932 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.528937:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d4f2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2260992->2273279) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178932 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016147.528985:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016147.528987:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 160 00000020:00010000:2.0:1734016147.528991:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016147.528992:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 160 00000100:00100000:2.0:1734016147.529069:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334655616:12345-192.168.201.152@tcp:106: Request processed in 174us (465us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016147.529075:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14838 00000080:00200000:2.0:1734016147.532996:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 400 00000020:00010000:2.0:1734016147.533004:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dc6d8 (ffff8800a72d9008/0) at cl_lock_release():233 00000020:00010000:2.0:1734016147.533008:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dc6d8 (ffff8800a72d9008/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.533015:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86764b0 (ffff8800a72d9f38/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.533020:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86764f0 (ffff88012db7c248/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.533023:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dc6d8 (ffff8800a72d9008/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.533025:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86764b0 (ffff8800a72d9008/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.533031:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86764f0 (ffff8800a72d9008/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016147.533109:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016147.533158:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016147.533167:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016147.533182:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016147.533204:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016147.533207:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e23f0, osc lock ffff8800aafcf0a0, flags 0x400001002 00000008:00010000:2.0:1734016147.533213:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016147.533288:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2430, osc lock ffff8800aafcf300, flags 0x400001002 00010000:00010000:2.0:1734016147.533313:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.533327:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.533339:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 7/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.533347:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 6/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.533381:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d500 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.533388:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d500 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.533450:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016147.533466:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332559872:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016147.537441:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334655744 00000100:00100000:2.0:1734016147.537454:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334655744 00000100:00100000:2.0:1734016147.537516:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14839 00000100:00100000:2.0:1734016147.537523:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ca80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334655744:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016147.537539:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d4f2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2260992->2273279) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178932 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.537551:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d4f2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2260992->2273279) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178932 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016147.537585:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016147.537588:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 160 00000020:00010000:2.0:1734016147.537591:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016147.537592:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 160 00000100:00100000:2.0:1734016147.537644:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ca80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334655744:12345-192.168.201.152@tcp:106: Request processed in 121us (295us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016147.537649:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14839 00010000:00010000:2.0:1734016147.539417:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d500 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.539431:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d500 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.539442:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d500 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.539449:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d500 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.539461:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d500 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016147.539473:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016147.539486:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d500 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.539493:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d500 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.539511:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d500 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.539519:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbc000 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016147.539524:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d500 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.539534:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016147.539539:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d500 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.539546:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d500 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.539582:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332559872:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016147.543283:0:9532:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016147.543300:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334655872 00000100:00100000:2.0:1734016147.543306:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334655872 00000100:00100000:2.0:1734016147.543348:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14840 00000100:00100000:2.0:1734016147.543356:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012b52e450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334655872:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016147.543372:0:9532:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d435 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 184320->192511) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d3178875 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.544054:0:9532:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016147.544062:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012b52e450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334655872:12345-192.168.201.152@tcp:104: Request processed in 706us (908us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016147.544068:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14840 00000100:00100000:2.0:1734016147.568763:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00010000:00010000:2.0:1734016147.645531:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d4f2 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2260992->2273279) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178932 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.645546:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d4f2 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2260992->2273279) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178932 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.645557:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d4f2 lrc: 5/1,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2260992->2273279) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178932 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.645565:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d4f2 lrc: 5/1,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2260992->2273279) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178932 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.645572:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d4f2 lrc: 4/1,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2260992->2273279) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178932 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016147.654599:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8e48 (ffff8800a72d9248/0) at cl_lock_release():233 00000020:00010000:2.0:1734016147.654607:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8e48 (ffff8800a72d9248/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.654614:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b35ba530 (ffff88012db7c6c8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016147.654626:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d507 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3375104->3383295) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178947 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.654637:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d507 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3375104->3383295) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178947 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016147.654654:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8e48 (ffff8800a72d9248/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.654659:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b35ba530 (ffff8800a72d9248/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016147.654674:0:10366:0:(vvp_io.c:533:vvp_io_advance()) advancing 4598 bytes 00000080:00200000:2.0:1734016147.654702:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016147.654707:0:10366:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 1 io complete with rc: 0, result: 4598, restart: 0 00000080:00200000:2.0:1734016147.654719:0:10366:0:(file.c:2044:ll_file_io_generic()) iot: 1, result: 4598 00000080:00200008:2.0:1734016147.654772:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016147.654784:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016147.654817:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016147.654837:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016147.654847:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016147.654863:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016147.654868:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d2c2 00000080:00010000:2.0:1734016147.654879:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2c2 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016147.654883:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016147.654891:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016147.654925:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016147.654945:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016147.654968:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016147.654971:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2db0, osc lock ffff8800aafcf300, flags 0x400001002 00000008:00010000:2.0:1734016147.654978:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016147.654979:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2df0, osc lock ffff8800aafcf0a0, flags 0x400001002 00010000:00010000:2.0:1734016147.654996:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d507 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3375104->3383295) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178947 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.655004:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d507 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3375104->3383295) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178947 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.655016:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d507 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3375104->3383295) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178947 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.655022:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d507 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3375104->3383295) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178947 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.655033:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d4f2 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2260992->2273279) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178932 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.655040:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d4f2 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2260992->2273279) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178932 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.655046:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d4f2 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2260992->2273279) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178932 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.655053:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d4f2 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2260992->2273279) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178932 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016147.655078:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 440 00000020:00010000:2.0:1734016147.655082:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_release():233 00000020:00010000:2.0:1734016147.655084:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.655087:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2db0 (ffff8800a72d97e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.655089:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2df0 (ffff88012db7c6c8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.655092:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.655094:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2db0 (ffff8800a72d9248/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.655098:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2df0 (ffff8800a72d9248/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016147.655106:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016147.655123:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016147.655126:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016147.655130:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016147.655134:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016147.655136:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2db0, osc lock ffff8800aafcf0a0, flags 0x400001002 00000008:00010000:2.0:1734016147.655138:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016147.655139:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2df0, osc lock ffff8800aafcf300, flags 0x400001002 00010000:00010000:2.0:1734016147.655144:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d507 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3375104->3383295) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178947 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.655150:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d507 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3375104->3383295) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178947 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.655158:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d507 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3375104->3383295) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178947 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.655164:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d507 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3375104->3383295) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178947 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.655174:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d4f2 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2260992->2273279) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178932 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.655181:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d4f2 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2260992->2273279) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178932 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.655189:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d4f2 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2260992->2273279) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178932 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.655193:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff8800a6c21d40/0x37cab3886ce5d4f2 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 2260992->2273279) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178932 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016147.655200:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 440 00000020:00010000:2.0:1734016147.655201:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_release():233 00000020:00010000:2.0:1734016147.655202:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.655204:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2db0 (ffff88012db7c6c8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.655205:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2df0 (ffff8800a72d97e8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.655207:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.655208:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2db0 (ffff8800a72d9248/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.655210:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2df0 (ffff8800a72d9248/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016147.655236:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016147.655243:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9625452 00000080:00200008:2.0:1734016147.655307:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 5278009, type: SEEK_SET 00000080:00200008:2.0:1734016147.655315:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 5278009, type: SEEK_SET, rc = 5278009 00000080:00200008:2.0:1734016147.655322:0:10366:0:(file.c:2290:do_file_read_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 5278009, count: 3421 00000080:00200000:2.0:1734016147.655327:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016147.655335:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016147.655369:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 1288 00000080:00200000:2.0:1734016147.655382:0:10366:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: read ppos: 5278009, bytes: 3421 00000080:00200000:2.0:1734016147.655387:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:2.0:1734016147.655396:0:10366:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b62c6a28[0x4, 1, [0x200000401:0xc:0x0] hash] 00000020:00200000:2.0:1734016147.655398:0:10366:0:(cl_io.c:195:cl_io_rw_init()) io range: 1 [5278009, 5281430) 0 0 00000080:00200000:2.0:1734016147.655400:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016147.655414:0:10366:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 1 chunk: [5278009, 5281430) 5281430, 3421 00020000:00200000:2.0:1734016147.655419:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016147.655438:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [1288, 1289] 00000008:00010000:2.0:1734016147.655447:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016147.655450:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8bb0, osc lock ffff8800aafcf300, flags 0x0 00010000:00010000:2.0:1734016147.655476:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d50e lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.655484:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d50e lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [1081344->1089535] (req 1081344->1089535) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.655518:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d50e lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [1081344->1089535] (req 1081344->1089535) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.655544:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8a80 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332560896:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016147.655592:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2600 going to sleep for 11 seconds 00000100:00100000:2.0:1734016147.730506:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000020:00010000:2.0:1734016147.742569:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8e48 (ffff8800a72d9248/0) at cl_lock_release():233 00000020:00010000:2.0:1734016147.742573:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8e48 (ffff8800a72d9248/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.742580:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8bb0 (ffff8800a72d97e8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016147.742592:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d50e lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1081344->1089535) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317894e expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.742603:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d50e lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1081344->1089535) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317894e expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016147.742621:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8e48 (ffff8800a72d9248/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.742623:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8bb0 (ffff8800a72d9248/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016147.742634:0:10366:0:(vvp_io.c:533:vvp_io_advance()) advancing 3421 bytes 00000080:00200000:2.0:1734016147.742655:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016147.742662:0:10366:0:(file.c:1964:ll_file_io_generic()) f16b.sanityn: 1 io complete with rc: 0, result: 3421, restart: 0 00000080:00200000:2.0:1734016147.742668:0:10366:0:(file.c:2044:ll_file_io_generic()) iot: 1, result: 3421 00000080:00200008:2.0:1734016147.742712:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016147.742725:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016147.742753:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016147.742780:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016147.742793:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016147.742811:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016147.742815:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d2c2 00000080:00010000:2.0:1734016147.742822:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2c2 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016147.742826:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016147.742831:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016147.742860:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016147.742873:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016147.742898:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016147.742901:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e20f0, osc lock ffff8800aafcf300, flags 0x400001002 00000008:00010000:2.0:1734016147.742907:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016147.742909:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2130, osc lock ffff8800aafcf0a0, flags 0x400001002 00010000:00010000:2.0:1734016147.742932:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d507 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3375104->3383295) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178947 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.742940:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d507 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3375104->3383295) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178947 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.742953:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d507 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3375104->3383295) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178947 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.742961:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d507 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3375104->3383295) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178947 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.742989:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d515 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.742997:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d515 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.746706:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d515 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.746726:0:1971:0:(ldlm_request.c:734:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d515 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xb46129d8d3178955 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.746737:0:1971:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d515 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d3178955 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.746746:0:1971:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d515 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xb46129d8d3178955 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016147.746760:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d515 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178955 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.746820:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d515 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d3178955 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.746827:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d515 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d3178955 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.746832:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d515 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb46129d8d3178955 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.746839:0:1971:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d515 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d3178955 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.746857:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332561664:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016147.750707:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334656512 00000100:00100000:2.0:1734016147.750724:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334656512 00000100:00100000:2.0:1734016147.750839:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14844 00000100:00100000:2.0:1734016147.750851:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334656512:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016147.750879:0:9532:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff8800a964d440/0x37cab3886ce5d507 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3375104->3383295) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d3178947 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.751322:0:9532:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016147.751330:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334656512:12345-192.168.201.152@tcp:104: Request processed in 480us (801us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016147.751342:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14844 00000100:00100000:2.0:1734016147.776572:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334656640 00000100:00100000:2.0:1734016147.776582:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334656640 00000100:00100000:2.0:1734016147.776628:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14845 00000100:00100000:2.0:1734016147.776638:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6472:x1818247334656640:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016147.776660:0:9532:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [188416->196607] (req 188416->196607) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.776672:0:9532:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [188416->196607] (req 188416->196607) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.776681:0:9532:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.776697:0:9532:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.776706:0:9532:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.776719:0:9532:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.776765:0:9532:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff8800a6c20fc0) 00000100:00100000:2.0:1734016147.776840:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6472:x1818247334656640:12345-192.168.201.152@tcp:105: Request processed in 203us (404us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016147.776849:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14845 00000100:00100000:2.0:1734016147.909747:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016147.918272:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 440 00000020:00010000:2.0:1734016147.918282:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_release():233 00000020:00010000:2.0:1734016147.918286:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.918293:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86766f0 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.918298:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676730 (ffff88012db7c3f8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.918303:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.918304:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86766f0 (ffff8800a72d9248/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.918310:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676730 (ffff8800a72d9248/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016147.918327:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016147.918368:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016147.918377:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016147.918393:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016147.918412:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016147.918416:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2cf0, osc lock ffff8800aafcfb50, flags 0x400001002 00000008:00010000:2.0:1734016147.918423:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016147.918424:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2d30, osc lock ffff8800aafcf8f0, flags 0x400001002 00010000:00010000:2.0:1734016147.918458:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd440/0x37cab3886ce5d52a lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.918466:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd440/0x37cab3886ce5d52a lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.919857:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbeac0/0x37cab3886ce5d515 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb46129d8d3178955 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.919884:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016147.919921:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332562432:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016147.923508:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016147.942565:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016147.949732:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 440 00000020:00010000:2.0:1734016147.949742:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_release():233 00000020:00010000:2.0:1734016147.949745:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.949753:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86764b0 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.949757:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86764f0 (ffff88012db7c3f8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.949763:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.949765:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86764b0 (ffff8800a72d9248/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.949773:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86764f0 (ffff8800a72d9248/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016147.949792:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016147.950014:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 1901 00000080:00200000:2.0:1734016147.950027:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016147.950042:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016147.950058:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1901, 1901] 00000008:00010000:2.0:1734016147.950073:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016147.950077:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8bb0, osc lock ffff8800aafcfa20, flags 0x0 00010000:00010000:2.0:1734016147.950091:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b6fb4800 type 11 mode 2 res 9663677440/7 (3592192 3596287) 00010000:00010000:2.0:1734016147.950103:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.950110:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [3592192->3596287] (req 3592192->3596287) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.950149:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [3592192->3596287] (req 3592192->3596287) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.950169:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8a80 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332562688:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016147.950270:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e26c0 going to sleep for 11 seconds 00000100:00100000:2.0:1734016147.958598:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016147.958687:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016147.959547:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14850 00000100:00100000:2.0:1734016147.959553:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73cb800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334657280:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016147.959562:0:4728:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d50e lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1081344->1089535) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d317894e expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.959601:0:4728:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:2.0:1734016147.959608:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73cb800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7692:x1818247334657280:12345-192.168.201.152@tcp:104: Request processed in 54us (923us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016147.959613:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14850 00000100:00100000:2.0:1734016147.965666:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334657408 00000100:00100000:2.0:1734016147.965676:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334657408 00000100:00100000:2.0:1734016147.965735:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14851 00000100:00100000:2.0:1734016147.965744:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7025:x1818247334657408:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016147.965763:0:4729:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [3592192->3596287] (req 3592192->3596287) gid 0 flags: 0x0 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.965776:0:4729:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [3592192->3596287] (req 3592192->3596287) gid 0 flags: 0x0 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.965785:0:4729:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 4/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x10000000000000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.965796:0:4729:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x10080000000000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.965806:0:4729:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x10080000000000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.965819:0:4729:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x80000000000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.965842:0:4729:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff88012dbbd200) 00000100:00100000:2.0:1734016147.965916:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7025:x1818247334657408:12345-192.168.201.152@tcp:105: Request processed in 171us (1318us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016147.965922:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14851 00000100:00100000:2.0:1734016147.976434:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8a80 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332562688:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016147.976481:0:10366:0:(ldlm_request.c:788:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.976533:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016147.976547:0:10366:0:(osc_lock.c:178:osc_lock_lvb_update()) ### acquired size 4194304, setting rss=4194304; kms=4194304, end=18446744073709551615 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x10020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.976559:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 5/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.976567:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.976718:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:2.0:1734016147.976741:0:1970:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e26c0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.976792:0:1970:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3592192 3596287) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016147.976835:0:1970:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73c8a80 - object 0x240000400:7 offset 3592192<>3596288 00000100:00100000:2.0:1734016147.976863:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332563200:192.168.201.152@tcp:3:fsx.0 00000008:00010000:2.0:1734016147.980440:0:1970:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e26c0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016147.980495:0:1970:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e26c0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016147.980521:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332563200:192.168.201.152@tcp:3:fsx.0 00000020:00010000:2.0:1734016147.989946:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8e48 (ffff8800a72d9248/0) at cl_lock_release():233 00000020:00010000:2.0:1734016147.989954:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8e48 (ffff8800a72d9248/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.989959:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f87b0 (ffff88012db7c3f8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016147.989972:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.989985:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016147.990015:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8e48 (ffff8800a72d9248/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.990018:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f87b0 (ffff8800a72d9248/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016147.990048:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016147.990074:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016147.990092:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016147.990113:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1902, 1902] 00000008:00010000:2.0:1734016147.990136:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016147.990140:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8c30, osc lock ffff8800aafcf690, flags 0x0 00010000:00010000:2.0:1734016147.990171:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.990185:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3596288 3600383) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.990198:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.990210:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.990273:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016147.990324:0:10366:0:(osc_cache.c:543:osc_extent_merge()) ### extent: ffff8800aa7e26c0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016147.990334:0:10366:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e26c0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016147.990342:0:10366:0:(osc_cache.c:560:osc_extent_merge()) ### extent: ffff8800a8676d80 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016147.990351:0:10366:0:(osc_cache.c:771:osc_extent_find()) ### extent: ffff8800a8676d80 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016147.990380:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af698248, count 1, dirtied 1 00000020:00010000:2.0:1734016147.990394:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8e48 (ffff8800a72d9248/0) at cl_lock_release():233 00000020:00010000:2.0:1734016147.990396:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8e48 (ffff8800a72d9248/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016147.990398:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8c30 (ffff88012db7c3f8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016147.990402:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.990410:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016147.990418:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8e48 (ffff8800a72d9248/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016147.990421:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8c30 (ffff8800a72d9248/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016147.990434:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016147.990506:0:10366:0:(file.c:5185:ll_fsync()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8), start 7786496, end 7794687, datasync 1 00000080:00200000:2.0:1734016147.990516:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016147.990522:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1734016147.990537:0:10366:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800a8676d80 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016147.990551:0:10366:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800a8676d80 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016147.991153:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3592192 3596287) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016147.991353:0:10366:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73c8a80 - object 0x240000400:7 offset 3592192<>3600384 00000100:00100000:2.0:1734016147.993565:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016147.993611:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332563584:192.168.201.152@tcp:4:fsx.0 00000008:00010000:2.0:1734016148.006822:0:1970:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800a8676d80 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.006914:0:1970:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800a8676d80 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.006944:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332563584:192.168.201.152@tcp:4:fsx.0 00000080:00200000:2.0:1734016148.031718:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 456 00000080:00200000:2.0:1734016148.031764:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016148.032655:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016148.032670:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016148.032705:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016148.032741:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016148.032755:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016148.032769:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016148.032772:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d2bb 00000080:00010000:2.0:1734016148.032781:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2bb from it ffff8800a7343cd8 00010000:00010000:2.0:1734016148.032783:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016148.032792:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016148.032826:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016148.032843:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016148.032875:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.032879:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e26f0, osc lock ffff8800aafcf690, flags 0x400001002 00000008:00010000:2.0:1734016148.032888:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.032890:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2730, osc lock ffff8800aafcee40, flags 0x400001002 00010000:00010000:2.0:1734016148.032907:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.032920:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.032936:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.032943:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.032980:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d53f lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.032986:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d53f lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.041028:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334657536 00000100:00100000:2.0:1734016148.041043:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334657536 00000100:00100000:2.0:1734016148.041775:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14852 00000100:00100000:2.0:1734016148.041786:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334657536:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016148.041807:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.041824:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.041870:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016148.041874:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 176 00000020:00010000:2.0:1734016148.041877:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016148.041879:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 176 00000100:00100000:2.0:1734016148.042990:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334657536:12345-192.168.201.152@tcp:106: Request processed in 1196us (2066us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016148.043001:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14852 00000100:00100000:2.0:1734016148.058090:0:1970:0:(client.c:2968:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8800b3cc4800: committing for last_committed 4294968284 gen 2 00010000:00010000:2.0:1734016148.058188:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d53f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.058306:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d53f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.058329:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d53f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.058339:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d53f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.058346:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d53f lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.058374:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016148.058418:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d53f lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.058425:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d53f lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.058434:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d53f lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.058443:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbc6c0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016148.058449:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d53f lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.058464:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016148.058473:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d53f lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.058479:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc6c0/0x37cab3886ce5d53f lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.058557:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332563840:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016148.066769:0:9532:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016148.066808:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334657664 00000100:00100000:2.0:1734016148.066822:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334657664 00000100:00100000:2.0:1734016148.066865:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14853 00000100:00100000:2.0:1734016148.066874:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012b52d850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334657664:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016148.066901:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.066913:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.066958:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016148.066960:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 176 00000020:00010000:2.0:1734016148.066963:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016148.066964:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 176 00000100:00100000:2.0:1734016148.067034:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012b52d850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334657664:12345-192.168.201.152@tcp:106: Request processed in 160us (1923us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016148.067042:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14853 00000080:00200000:2.0:1734016148.074364:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 456 00000020:00010000:2.0:1734016148.074377:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.074380:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.074389:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676db0 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.074396:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676df0 (ffff88012db7c3f8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.074399:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.074402:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676db0 (ffff8800a72d9248/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.074410:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676df0 (ffff8800a72d9248/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.074428:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016148.074452:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9625452 00000080:00200008:2.0:1734016148.074495:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016148.074515:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016148.074551:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016148.074584:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016148.074605:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016148.074632:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016148.074636:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d2bb 00000080:00010000:2.0:1734016148.074644:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2bb from it ffff8800a7343cd8 00010000:00010000:2.0:1734016148.074647:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016148.074656:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016148.074694:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016148.074712:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016148.074750:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.074754:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e26f0, osc lock ffff8800aafced10, flags 0x400001002 00000008:00010000:2.0:1734016148.074763:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.074764:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2730, osc lock ffff8800aafcfdb0, flags 0x400001002 00010000:00010000:2.0:1734016148.074787:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.074797:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.074808:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.074817:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.074861:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d54d lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.074870:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d54d lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.074983:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016148.075012:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332564096:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016148.081383:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334657792 00000100:00100000:2.0:1734016148.081398:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334657792 00000100:00100000:2.0:1734016148.081451:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14854 00000100:00100000:2.0:1734016148.081461:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334657792:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016148.081480:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.081502:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.081537:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016148.081540:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 176 00000020:00010000:2.0:1734016148.081544:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016148.081545:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 176 00000100:00100000:2.0:1734016148.081607:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334657792:12345-192.168.201.152@tcp:106: Request processed in 147us (564us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016148.081615:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14854 00010000:00010000:2.0:1734016148.087980:0:1970:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d54d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.088010:0:1970:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d54d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.088027:0:1970:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d54d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.088034:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d54d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.088043:0:1970:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d54d lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.088059:0:1970:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016148.088936:0:1970:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d54d lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.088950:0:1970:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d54d lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.088958:0:1970:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d54d lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.088965:0:1970:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbeac0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016148.088972:0:1970:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d54d lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.088990:0:1970:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016148.088996:0:1970:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d54d lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.089005:0:1970:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d54d lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.089071:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332564096:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016148.093273:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334657920 00000100:00100000:2.0:1734016148.093293:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334657920 00000100:00100000:2.0:1734016148.093389:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14855 00000100:00100000:2.0:1734016148.093401:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334657920:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016148.093431:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.093447:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.093504:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016148.093508:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 176 00000020:00010000:2.0:1734016148.093512:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016148.093513:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 176 00000100:00100000:2.0:1734016148.093603:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334657920:12345-192.168.201.152@tcp:106: Request processed in 205us (603us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016148.093610:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14855 00000080:00200000:2.0:1734016148.098586:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 456 00000020:00010000:2.0:1734016148.098598:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.098602:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.098611:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676030 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.098616:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676070 (ffff88012db7c3f8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.098619:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.098620:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676030 (ffff8800a72d9248/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.098627:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676070 (ffff8800a72d9248/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.098644:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016148.098664:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9625452 00000080:00200008:2.0:1734016148.098697:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016148.098706:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016148.098737:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016148.098769:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016148.098782:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016148.098806:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016148.098810:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d2bb 00000080:00010000:2.0:1734016148.098817:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2bb from it ffff8800a7343cd8 00010000:00010000:2.0:1734016148.098820:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016148.098829:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016148.098862:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016148.098876:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016148.098904:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.098908:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2330, osc lock ffff8800aafce980, flags 0x400001002 00000008:00010000:2.0:1734016148.098915:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.098918:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2370, osc lock ffff8800aafcf560, flags 0x400001002 00010000:00010000:2.0:1734016148.098936:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.098946:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.098958:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.098965:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.098998:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d55b lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.099007:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d55b lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.099127:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016148.099148:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332564352:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016148.104738:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334658048 00000100:00100000:2.0:1734016148.104757:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334658048 00000100:00100000:2.0:1734016148.104834:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14856 00000100:00100000:2.0:1734016148.104848:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334658048:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016148.104872:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.104888:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.104932:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016148.104936:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 176 00000020:00010000:2.0:1734016148.104939:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016148.104940:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 176 00000100:00100000:2.0:1734016148.105023:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334658048:12345-192.168.201.152@tcp:106: Request processed in 178us (546us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016148.105031:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14856 00010000:00010000:2.0:1734016148.109208:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d55b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.109267:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d55b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.109288:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d55b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.109296:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d55b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.109311:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d55b lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.109332:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016148.109555:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d55b lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.109561:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d55b lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.109568:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d55b lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.109575:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbeac0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016148.109581:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d55b lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.109593:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016148.109600:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d55b lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.109605:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d55b lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.110450:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332564352:192.168.201.152@tcp:101:fsx.0 00000080:00200000:2.0:1734016148.119449:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 456 00000020:00010000:2.0:1734016148.119458:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.119462:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.119469:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676270 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.119473:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86762b0 (ffff88012db7c3f8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.119476:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.119478:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676270 (ffff8800a72d9248/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.119494:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86762b0 (ffff8800a72d9248/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.119771:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016148.119789:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9625452 00000080:00200008:2.0:1734016148.119814:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016148.119822:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016148.119845:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016148.119861:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016148.119872:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016148.119887:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016148.119890:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d2bb 00000080:00010000:2.0:1734016148.119895:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2bb from it ffff8800a7343cd8 00010000:00010000:2.0:1734016148.119898:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016148.119905:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016148.119933:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016148.119944:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016148.119964:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.119966:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2db0, osc lock ffff8800aafcfc80, flags 0x400001002 00000008:00010000:2.0:1734016148.119971:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.119972:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2df0, osc lock ffff8800aafce720, flags 0x400001002 00010000:00010000:2.0:1734016148.119985:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.119993:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.120002:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.120008:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.120027:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d569 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.120032:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d569 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.124177:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334658304 00000100:00100000:2.0:1734016148.124196:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334658304 00000100:00100000:2.0:1734016148.124410:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14858 00000100:00100000:2.0:1734016148.124423:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334658304:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016148.124448:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.124464:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.124518:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016148.124521:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 176 00000020:00010000:2.0:1734016148.124524:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016148.124525:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 176 00000100:00100000:2.0:1734016148.124593:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334658304:12345-192.168.201.152@tcp:106: Request processed in 174us (753us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016148.124602:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14858 00010000:00010000:2.0:1734016148.129246:0:1971:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d569 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.129273:0:1971:0:(ldlm_request.c:635:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d569 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.129286:0:1971:0:(ldlm_request.c:552:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d569 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.129291:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d569 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.129299:0:1971:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d569 lrc: 5/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.129320:0:1971:0:(osc_lock.c:178:osc_lock_lvb_update()) ### no dlm lock: acquired size 4194304, setting rss=4194304; leaving kms=0, end=18446744073709551615 00010000:00010000:2.0:1734016148.129715:0:1971:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d569 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.129722:0:1971:0:(ldlm_lock.c:877:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d569 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.129730:0:1971:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d569 lrc: 4/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.129736:0:1971:0:(ldlm_lockd.c:1942:ldlm_handle_bl_callback()) Lock ffff88012dbbeac0 already unused, calling callback (ffffffffa0a54d90) 00010000:00010000:2.0:1734016148.129741:0:1971:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d569 lrc: 5/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.129753:0:1971:0:(ldlm_request.c:1296:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1734016148.129758:0:1971:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d569 lrc: 3/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.129763:0:1971:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbeac0/0x37cab3886ce5d569 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.130473:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332564608:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016148.133897:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334658432 00000100:00100000:2.0:1734016148.133912:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334658432 00000100:00100000:2.0:1734016148.134014:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14859 00000100:00100000:2.0:1734016148.134027:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334658432:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016148.134053:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.134070:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.134112:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016148.134115:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 176 00000020:00010000:2.0:1734016148.134118:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016148.134119:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016147 atime: 1734016147 ctime: 1734016147 blocks: 176 00000100:00100000:2.0:1734016148.134205:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334658432:12345-192.168.201.152@tcp:106: Request processed in 180us (632us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016148.134214:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14859 00000080:00200000:2.0:1734016148.138625:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 456 00000020:00010000:2.0:1734016148.138634:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.138638:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.138644:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676270 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.138648:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86762b0 (ffff88012db7c3f8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.138651:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dcdb0 (ffff8800a72d9248/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.138653:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676270 (ffff8800a72d9248/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.138657:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86762b0 (ffff8800a72d9248/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.138671:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016148.138686:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9625452 00000080:00200008:2.0:1734016148.138759:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 1885953, type: SEEK_SET 00000080:00200008:2.0:1734016148.138767:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 1885953, type: SEEK_SET, rc = 1885953 00000080:00200008:2.0:1734016148.138778:0:10366:0:(file.c:2466:do_file_write_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 1885953, count: 3732 00000080:00200000:2.0:1734016148.138784:0:10366:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: write ppos: 1885953, bytes: 3732 00000080:00200000:2.0:1734016148.138790:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:2.0:1734016148.138798:0:10366:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b6e5c708[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00200000:2.0:1734016148.138800:0:10366:0:(cl_io.c:195:cl_io_rw_init()) io range: 2 [1885953, 1889685) 0 0 00000080:00200000:2.0:1734016148.138805:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016148.138821:0:10366:0:(file.c:1893:ll_file_io_generic()) Range lock [460, 461] 00020000:00200000:2.0:1734016148.138832:0:10366:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 0 chunk: [1885953, 1889685) 1889685, 3732 00020000:00200000:2.0:1734016148.138837:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016148.138857:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [460, 461] 00000008:00010000:2.0:1734016148.138873:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016148.138877:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8cb0, osc lock ffff8800aafce5f0, flags 0x0 00010000:00010000:2.0:1734016148.138914:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.138922:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1884160->1892351] (req 1884160->1892351) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.138968:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1884160->1892351] (req 1884160->1892351) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.138993:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8380 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332564864:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016148.139745:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e29c0 going to sleep for 11 seconds 00000100:00100000:2.0:1734016148.143036:0:4729:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016148.143057:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334658560 00000100:00100000:2.0:1734016148.143068:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334658560 00000100:00100000:2.0:1734016148.143118:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14860 00000100:00100000:2.0:1734016148.143124:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a703b850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334658560:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016148.143141:0:4729:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.144360:0:4729:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016148.144372:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a703b850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10116:x1818247334658560:12345-192.168.201.152@tcp:104: Request processed in 1245us (1445us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016148.144379:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14860 00000100:00100000:2.0:1734016148.147035:0:10366:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8380 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332564864:192.168.201.152@tcp:101:fsx.0 00010000:00010000:2.0:1734016148.147082:0:10366:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1884160->1892351] (req 1884160->1892351) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31789b7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.147099:0:10366:0:(ldlm_request.c:273:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1884160->1892351] (req 1884160->1892351) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31789b7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.147107:0:10366:0:(ldlm_request.c:302:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1884160->1892351] (req 1884160->1892351) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31789b7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016148.154704:0:4736:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016148.160240:0:4736:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1734016148.160341:0:4736:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 2/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.160355:0:4736:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1734016148.160942:0:4736:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 1/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.160950:0:4736:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c20fc0/0x37cab3886ce5d51c lrc: 0/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 188416->196607) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d317895c expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.160995:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016148.161027:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332565120:192.168.201.152@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1734016148.162953:0:4736:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016148.162961:0:4736:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016148.162966:0:4736:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1734016148.168928:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334658688 00000100:00100000:2.0:1734016148.168959:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334658688 00000100:00100000:2.0:1734016148.169026:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14861 00000100:00100000:2.0:1734016148.169035:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7025:x1818247334658688:12345-192.168.201.152@tcp:105: 00010000:00010000:2.0:1734016148.169056:0:4729:0:(ldlm_lockd.c:2529:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1884160->1892351] (req 1884160->1892351) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31789b7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.169071:0:4729:0:(ldlm_lockd.c:1987:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [1884160->1892351] (req 1884160->1892351) gid 0 flags: 0x20000 nid: local remote: 0xb46129d8d31789b7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.169082:0:4729:0:(ldlm_lockd.c:2064:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1884160->1892351) gid 0 flags: 0x10000000020000 nid: local remote: 0xb46129d8d31789b7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.169095:0:4729:0:(ldlm_lock.c:706:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1884160->1892351) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31789b7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.169105:0:4729:0:(ldlm_resource.c:1684:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1884160->1892351) gid 0 flags: 0x10080000020000 nid: local remote: 0xb46129d8d31789b7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.169119:0:4729:0:(ldlm_lockd.c:2092:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1884160->1892351) gid 0 flags: 0x80000020000 nid: local remote: 0xb46129d8d31789b7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.169162:0:4729:0:(ldlm_lockd.c:2103:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff88012dbbc6c0) 00000100:00100000:2.0:1734016148.169253:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7025:x1818247334658688:12345-192.168.201.152@tcp:105: Request processed in 220us (2537us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016148.169261:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14861 00000100:00100000:2.0:1734016148.179359:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332565120:192.168.201.152@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1734016148.204032:0:10366:0:(rw26.c:1006:ll_write_end()) pos 1885953, len 2303, copied 2303 00000080:00200000:2.0:1734016148.204048:0:10366:0:(rw26.c:1044:ll_write_end()) page@ffff88012b465438[3 ffff8800b6e5c788 2 1 ffff8800a70dce10] 00000080:00200000:2.0:1734016148.204051:0:10366:0:(rw26.c:1044:ll_write_end()) vmpage @ffffea0002a02dc0 1fffff0000082d 2:0 ffff88012b465438 460 lru 00000080:00200000:2.0:1734016148.204060:0:10366:0:(rw26.c:1044:ll_write_end()) osc-page@ffff88012b465498 460: 1< 1 - - > 2< 1884160 0 4096 0x0 0x40009 | ffff8800a947dcc8 ffff8800a86f2b60 > 3< 0 0 > 4< 0 0 8 489881600 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1734016148.204062:0:10366:0:(rw26.c:1044:ll_write_end()) end page@ffff88012b465438 00000080:00200000:2.0:1734016148.204064:0:10366:0:(rw26.c:1044:ll_write_end()) queued page: 1. 00000080:00200000:2.0:1734016148.204071:0:10366:0:(rw26.c:819:ll_write_begin()) Writing 461 of 0 to 1429 bytes 00000100:00100000:2.0:1734016148.204277:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:2.0:1734016148.204294:0:1970:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e20c0 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1884160->1892351) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789b7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.204354:0:1970:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1888256 1892351) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1884160->1892351) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789b7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016148.204418:0:1970:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73c8000 - object 0x280000400:7 offset 1888256<>1892352 00000100:00100000:2.0:1734016148.204449:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332565632:192.168.201.152@tcp:3:fsx.0 00000008:00010000:2.0:1734016148.208688:0:1970:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e20c0 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1884160->1892351) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789b7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.208727:0:1970:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e20c0 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1884160->1892351) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789b7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.208747:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332565632:192.168.201.152@tcp:3:fsx.0 00000100:00100000:2.0:1734016148.222454:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334658816 00000100:00100000:2.0:1734016148.222471:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334658816 00000100:00100000:2.0:1734016148.222561:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14862 00000100:00100000:2.0:1734016148.222573:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334658816:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016148.222600:0:9532:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1884160->1892351) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d31789b7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.222714:0:9532:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016148.222723:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334658816:12345-192.168.201.152@tcp:104: Request processed in 152us (783us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016148.222732:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14862 00000080:00200000:2.0:1734016148.264830:0:5236:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016148.266671:0:5236:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1734016148.266841:0:5236:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 2/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1884160->1892351) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d31789b7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.266852:0:5236:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1734016148.266892:0:5236:0:(ldlm_lockd.c:1952:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1884160->1892351) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d31789b7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.266900:0:5236:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbc6c0/0x37cab3886ce5d577 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1884160->1892351) gid 0 flags: 0x804c69400020020 nid: local remote: 0xb46129d8d31789b7 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.266929:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016148.266932:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016148.266937:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1734016148.266958:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016148.266988:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332566016:192.168.201.152@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1734016148.284581:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000020:00010000:2.0:1734016148.299142:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8e48 (ffff8800a72d9248/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.299148:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8e48 (ffff8800a72d9248/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.299159:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b63f8430 (ffff88012db7c3f8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016148.299182:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.299208:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.300141:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8e48 (ffff8800a72d9248/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.300147:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b63f8430 (ffff8800a72d9248/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.300245:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016148.300315:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016148.300327:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016148.300368:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016148.300397:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016148.300413:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016148.300435:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016148.300438:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d2bb 00000080:00010000:2.0:1734016148.300446:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2bb from it ffff8800a7343cd8 00010000:00010000:2.0:1734016148.300449:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016148.300455:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016148.300494:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016148.300511:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016148.300538:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.300542:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e20f0, osc lock ffff8800aafcf7c0, flags 0x400001002 00000008:00010000:2.0:1734016148.300550:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.300552:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2130, osc lock ffff8800aafceab0, flags 0x400001002 00010000:00010000:2.0:1734016148.300576:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.300586:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.300605:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.300613:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.300653:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d585 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.300662:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd440/0x37cab3886ce5d585 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.300795:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016148.300830:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332566272:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016148.316015:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016148.329694:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 496 00000020:00010000:2.0:1734016148.329718:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.329724:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.329731:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e20f0 (ffff88012db7c098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.329742:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2130 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.329744:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.329747:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e20f0 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.329761:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2130 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.329805:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016148.329848:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016148.329859:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016148.329871:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016148.329897:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.329901:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e20f0, osc lock ffff8800aafceab0, flags 0x400001002 00000008:00010000:2.0:1734016148.329910:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.329911:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2130, osc lock ffff8800aafcf7c0, flags 0x400001002 00010000:00010000:2.0:1734016148.329943:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.329968:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.329987:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.329999:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.330750:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b582d000 type 11 mode 6 res 9663677440/7 (0 18446744073709551615) 00010000:00010000:2.0:1734016148.330773:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d58c lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.330782:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbd680/0x37cab3886ce5d58c lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.330903:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016148.330942:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332566400:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016148.336451:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016148.336508:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334659200 00000100:00100000:2.0:1734016148.336523:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334659200 00000100:00100000:2.0:1734016148.336595:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14865 00000100:00100000:2.0:1734016148.336608:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73c9c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334659200:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016148.336629:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.336645:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.336690:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b6e5c708[0x4, 3, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016148.336693:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016148 atime: 1734016147 ctime: 1734016148 blocks: 176 00000020:00010000:2.0:1734016148.336697:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800a96e8ed8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016148.336698:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016148 atime: 1734016147 ctime: 1734016148 blocks: 176 00000100:00100000:2.0:1734016148.336783:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73c9c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:10116:x1818247334659200:12345-192.168.201.152@tcp:106: Request processed in 177us (333us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016148.336792:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14865 00000080:00200000:2.0:1734016148.343571:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 496 00000020:00010000:2.0:1734016148.343589:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.343595:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.343606:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e20f0 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.343616:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2130 (ffff88012db7c098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.343619:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.343622:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e20f0 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.343630:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2130 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.343664:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016148.343686:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9625452 00000080:00a00000:2.0:1734016148.343826:0:10366:0:(llite_mmap.c:611:ll_file_mmap()) VFS_Op: fid=[0x200000401:0xc:0x0] vma=ffff8800a9182798 start=0x7fef68fb6000 end=0x7fef68fb9000 vm_flags=0xfb 00000080:00200000:2.0:1734016148.343846:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016148.343860:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016148.343883:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.343888:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e20f0, osc lock ffff8800aafcf7c0, flags 0x400001002 00000008:00010000:2.0:1734016148.343894:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.343895:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2130, osc lock ffff8800aafceab0, flags 0x400001002 00010000:00010000:2.0:1734016148.343928:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.343948:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.343965:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.343974:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.344024:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d593 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.344032:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbc000/0x37cab3886ce5d593 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.344161:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016148.344200:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332566528:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016148.349792:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016148.367380:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 496 00000020:00010000:2.0:1734016148.367399:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.367404:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.367412:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e20f0 (ffff88012db7c098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.367420:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2130 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.367424:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.367426:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e20f0 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.367437:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2130 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.367472:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016148.367595:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 2075 00000080:00200000:2.0:1734016148.367615:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016148.367641:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016148.367660:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [2075, 2075] 00000008:00010000:2.0:1734016148.367691:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016148.367695:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef81b0, osc lock ffff8800aafceab0, flags 0x0 00010000:00010000:2.0:1734016148.367728:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.367748:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4304896 4308991) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.367762:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.367770:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.367779:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.367846:0:10366:0:(ldlm_lock.c:1516:ldlm_lock_match_with_skip()) ### not matched ns ffff8800b582d000 type 11 mode 70 res 9663677440/7 (0 4095) 00000100:00100000:2.0:1734016148.367926:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:2.0:1734016148.367948:0:1970:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e20c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.367977:0:1970:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4304896 4308991) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016148.368023:0:1970:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73caa00 - object 0x280000400:7 offset 4304896<>4308992 00000100:00100000:2.0:1734016148.368051:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332566784:192.168.201.152@tcp:3:fsx.0 00000020:00010000:2.0:1734016148.378206:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.378213:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.378307:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef81b0 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016148.378323:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.378338:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.378361:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.378363:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef81b0 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.378400:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016148.378426:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016148.378449:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016148.378474:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [2075, 2075] 00000008:00010000:2.0:1734016148.378501:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016148.378508:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef81b0, osc lock ffff8800aafceab0, flags 0x0 00010000:00010000:2.0:1734016148.378539:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.378557:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4304896 4308991) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.378570:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.378577:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 6/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.378584:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.378639:0:10366:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800aa7e2600 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016148.378665:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 1, dirtied 1 00000020:00010000:2.0:1734016148.378683:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.378687:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.378689:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef81b0 (ffff88012db7c098/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016148.378694:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.378700:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.378710:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.378711:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef81b0 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.378717:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016148.378794:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 2076 00000080:00200000:2.0:1734016148.378803:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016148.378808:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016148.378812:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [2076, 2076] 00000008:00010000:2.0:1734016148.378817:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016148.378820:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef81b0, osc lock ffff8800aafceab0, flags 0x0 00010000:00010000:2.0:1734016148.378827:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.378840:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4308992 4313087) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.378847:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.378852:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.378858:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.378991:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:2.0:1734016148.379010:0:1970:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e23c0 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.379041:0:1970:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4308992 4313087) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016148.379093:0:1970:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73c9880 - object 0x280000400:7 offset 4308992<>4313088 00000100:00100000:2.0:1734016148.379124:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332567040:192.168.201.152@tcp:3:fsx.0 00000020:00010000:2.0:1734016148.386848:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.386854:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.386858:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef81b0 (ffff88012db7c098/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016148.386869:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.386881:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.386900:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.386903:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef81b0 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.386931:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016148.386953:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016148.386973:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016148.386989:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [2076, 2076] 00000008:00010000:2.0:1734016148.387007:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016148.387012:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef81b0, osc lock ffff8800aafceab0, flags 0x0 00010000:00010000:2.0:1734016148.387034:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.387048:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4308992 4313087) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.387062:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.387070:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.387076:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.387130:0:10366:0:(osc_cache.c:543:osc_extent_merge()) ### extent: ffff8800aa7e2900 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.387140:0:10366:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2900 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.387148:0:10366:0:(osc_cache.c:560:osc_extent_merge()) ### extent: ffff8800aa7e2600 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.387157:0:10366:0:(osc_cache.c:771:osc_extent_find()) ### extent: ffff8800aa7e2600 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016148.387179:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 1, dirtied 1 00000020:00010000:2.0:1734016148.387193:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.387196:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.387200:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef81b0 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016148.387203:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.387210:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.387244:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.387246:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef81b0 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.387254:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016148.387322:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 2077 00000080:00200000:2.0:1734016148.387331:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016148.387338:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016148.387342:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [2077, 2077] 00000008:00010000:2.0:1734016148.387347:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016148.387349:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef81b0, osc lock ffff8800aafceab0, flags 0x0 00010000:00010000:2.0:1734016148.387355:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.387362:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4313088 4317183) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.387370:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.387377:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.387383:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.387507:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00010000:2.0:1734016148.387517:0:1970:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e2900 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.387551:0:1970:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4313088 4317183) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016148.387607:0:1970:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73ca680 - object 0x280000400:7 offset 4313088<>4317184 00000100:00100000:2.0:1734016148.387640:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332567296:192.168.201.152@tcp:3:fsx.0 00000020:00010000:2.0:1734016148.391596:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.391601:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.391607:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef81b0 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016148.391617:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.391630:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.391649:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.391651:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef81b0 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.392333:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016148.392363:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016148.392384:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016148.392402:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [2077, 2077] 00000008:00010000:2.0:1734016148.392424:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016148.392437:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef81b0, osc lock ffff8800aafceab0, flags 0x0 00010000:00010000:2.0:1734016148.392467:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.392491:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4313088 4317183) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.392502:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.392507:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 7/0,2 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.392512:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.392562:0:10366:0:(osc_cache.c:543:osc_extent_merge()) ### extent: ffff8800aa7e2900 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.392569:0:10366:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2900 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.392575:0:10366:0:(osc_cache.c:560:osc_extent_merge()) ### extent: ffff8800aa7e2600 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.392583:0:10366:0:(osc_cache.c:771:osc_extent_find()) ### extent: ffff8800aa7e2600 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016148.392602:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af65b248, count 1, dirtied 1 00000020:00010000:2.0:1734016148.392613:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.392615:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.392616:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef81b0 (ffff88012db7c098/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016148.392619:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.392625:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.392634:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.392636:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef81b0 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.392643:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016148.392688:0:10366:0:(file.c:5185:ll_fsync()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8), start 8499200, end 8511487, datasync 1 00000080:00200000:2.0:1734016148.392698:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016148.392703:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1734016148.392717:0:10366:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e2600 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.392728:0:10366:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800aa7e2600 ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.392912:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (4304896 4308991) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016148.393008:0:10366:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73ca300 - object 0x280000400:7 offset 4304896<>4317184 00000080:00200000:2.0:1734016148.411868:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016148.411897:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016148.411911:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1734016148.412012:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016148.412026:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332567552:192.168.201.152@tcp:16:fsx.0 00000080:00200000:2.0:1734016148.432633:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 520 00000080:00200000:2.0:1734016148.432673:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016148.432805:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016148.432819:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016148.432865:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016148.432901:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016148.432921:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016148.432950:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016148.432956:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d2c2 00000080:00010000:2.0:1734016148.432970:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2c2 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016148.432975:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016148.432985:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016148.433019:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016148.433041:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016148.433077:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.433082:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2630, osc lock ffff8800aafceab0, flags 0x400001002 00000008:00010000:2.0:1734016148.433090:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.433091:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2670, osc lock ffff8800aafcf7c0, flags 0x400001002 00010000:00010000:2.0:1734016148.433138:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d59a lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.433148:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbdd40/0x37cab3886ce5d59a lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.434653:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.434667:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.434685:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.434697:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.445076:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016148.458937:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016148.471785:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016148.476754:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 520 00000020:00010000:2.0:1734016148.476766:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.476769:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.476777:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86766f0 (ffff88012db7c098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.476782:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676730 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.476785:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.476787:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86766f0 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.476792:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676730 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.476807:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016148.476892:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 377 00000080:00200000:2.0:1734016148.476905:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016148.476921:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016148.476939:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [377, 377] 00000008:00010000:2.0:1734016148.476957:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016148.476961:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8c30, osc lock ffff8800aafcef70, flags 0x0 00010000:00010000:2.0:1734016148.476985:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.477002:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1544192 1548287) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.477017:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 6/1,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.477024:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 6/1,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.477030:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.488773:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.488778:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.488783:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8c30 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016148.488798:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 4/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.488812:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.488838:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.488841:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8c30 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.488877:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016148.488980:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 378 00000080:00200000:2.0:1734016148.488991:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016148.489009:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016148.489028:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 0 [378, 378] 00000008:00010000:2.0:1734016148.489043:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016148.489050:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8c30, osc lock ffff8800aafcef70, flags 0x0 00010000:00010000:2.0:1734016148.489078:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.489092:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1548288 1552383) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.489106:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 6/1,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.489111:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 6/1,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.489119:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.504843:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.504861:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.504865:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8c30 (ffff88012db7c098/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016148.504878:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 4/1,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.504895:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.504920:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.504923:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8c30 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.504964:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016148.505077:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af65b0c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016148.505090:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af65b0c8),name=f16b.sanityn 00000002:00010000:2.0:1734016148.505122:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016148.505149:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016148.505162:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016148.505180:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) 00000080:00010000:2.0:1734016148.505184:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af65b0c8) for lock 0x37cab3886ce5d2bb 00000080:00010000:2.0:1734016148.505192:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2bb from it ffff8800a7343cd8 00010000:00010000:2.0:1734016148.505197:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016148.505205:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b3cc4800 lock: ffff88012dbbf180/0x37cab3886ce5d2bb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d31786f4 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016148.505310:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016148.505326:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016148.505353:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.505358:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f30, osc lock ffff8800aafcef70, flags 0x400001002 00000008:00010000:2.0:1734016148.505368:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.505369:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f70, osc lock ffff8800aafce4c0, flags 0x400001002 00010000:00010000:2.0:1734016148.505388:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.505397:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.505414:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.505422:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.505456:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d5af lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.505464:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcfc0/0x37cab3886ce5d5af lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.512447:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016148.519307:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 520 00000020:00010000:2.0:1734016148.519316:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.519321:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.519326:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f30 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.519332:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f70 (ffff88012db7c098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.519335:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.519337:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f30 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.519345:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f70 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.519367:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016148.519395:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016148.519403:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016148.519420:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016148.519448:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.519452:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f30, osc lock ffff8800aafce4c0, flags 0x400001002 00000008:00010000:2.0:1734016148.519459:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.519461:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f70, osc lock ffff8800aafcef70, flags 0x400001002 00010000:00010000:2.0:1734016148.519499:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.519513:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.519524:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.519533:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.519575:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d5b6 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.519582:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbdf80/0x37cab3886ce5d5b6 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.532140:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016148.554769:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 520 00000020:00010000:2.0:1734016148.554784:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.554788:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.554795:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f30 (ffff88012db7c098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.554802:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f70 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.554804:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.554806:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f30 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.554817:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f70 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.554843:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016148.554863:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9625452 00000080:00200008:2.0:1734016148.554897:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016148.554907:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016148.554944:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016148.554975:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016148.554993:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016148.555009:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016148.555013:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d2c2 00000080:00010000:2.0:1734016148.555020:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2c2 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016148.555024:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016148.555033:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016148.555064:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016148.555078:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016148.555105:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.555108:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f30, osc lock ffff8800aafcef70, flags 0x400001002 00000008:00010000:2.0:1734016148.555114:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.555115:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f70, osc lock ffff8800aafce4c0, flags 0x400001002 00010000:00010000:2.0:1734016148.555142:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d5bd lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.555155:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbcd80/0x37cab3886ce5d5bd lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.555287:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.555294:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.555304:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.555310:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.562361:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016148.591379:0:9532:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016148.591409:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334660224 00000100:00100000:2.0:1734016148.591421:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334660224 00000100:00100000:2.0:1734016148.591503:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14873 00000100:00100000:2.0:1734016148.591511:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff88012b52c850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334660224:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016148.591528:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.591541:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.591581:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 6, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016148.591583:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 344 00000020:00010000:2.0:1734016148.591586:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016148.591587:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 344 00000100:00100000:2.0:1734016148.591648:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff88012b52c850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334660224:12345-192.168.201.152@tcp:106: Request processed in 139us (2018us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016148.591654:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14873 00000080:00200000:2.0:1734016148.599867:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 520 00000020:00010000:2.0:1734016148.599887:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.599891:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.599900:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676330 (ffff88012db7c098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.599908:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676370 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.599916:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.599918:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676330 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.599930:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676370 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.599954:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016148.599971:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9625452 00000080:00200000:2.0:1734016148.600068:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000080:00200000:2.0:1734016148.600077:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:2.0:1734016148.600096:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016148.600113:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [2009, 2009] 00000008:00010000:2.0:1734016148.600131:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016148.600135:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8c30, osc lock ffff8800aafcf1d0, flags 0x0 00010000:00010000:2.0:1734016148.600178:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.600188:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [4034560->4038655] (req 4034560->4038655) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.600254:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [4034560->4038655] (req 4034560->4038655) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.600289:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job fsx:f1fa65c5-ac21-42fc-9fc8-6a110334833e:10366:1818247332569088:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016148.600375:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2d80 going to sleep for 11 seconds 00000100:00100000:2.0:1734016148.619286:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334660352 00000100:00100000:2.0:1734016148.619299:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334660352 00000100:00100000:2.0:1734016148.619362:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14874 00000100:00100000:2.0:1734016148.619371:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800b341c700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334660352:12345-192.168.201.152@tcp:104: 00010000:00010000:2.0:1734016148.619392:0:4729:0:(ldlm_lockd.c:2517:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d538 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 3592192->3596287) gid 0 flags: 0x800420000020000 nid: local remote: 0xb46129d8d3178978 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.619499:0:4729:0:(ldlm_lockd.c:2208:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1734016148.619507:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800b341c700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:7692:x1818247334660352:12345-192.168.201.152@tcp:104: Request processed in 136us (374us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016148.619515:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14874 00000100:00100000:2.0:1734016148.631082:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000020:00010000:2.0:1734016148.654618:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.654631:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.654639:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8c30 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016148.654661:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.654682:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.654720:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a71a8308 (ffff8800a72d97e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.654723:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8c30 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.654765:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016148.654816:0:10366:0:(file.c:5861:ll_getattr_dentry()) START file f16b.sanityn:[0x200000401:0xc:0x0](ffff8800af6980c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1734016148.654827:0:10366:0:(file.c:5757:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8),name=f16b.sanityn 00000002:00010000:2.0:1734016148.654869:0:10366:0:(mdc_locks.c:1411:mdc_intent_lock()) (name: ,[0x200000401:0xc:0x0]) in obj [0x200000401:0xc:0x0], intent: getattr flags 0 00010000:00010000:2.0:1734016148.654903:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016148.654923:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1734016148.654952:0:10366:0:(dcache.c:297:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) 00000080:00010000:2.0:1734016148.654956:0:10366:0:(llite_internal.h:2012:ll_set_lock_data()) setting l_data to inode [0x200000401:0xc:0x0](ffff8800af6980c8) for lock 0x37cab3886ce5d2c2 00000080:00010000:2.0:1734016148.654966:0:10366:0:(dcache.c:194:ll_intent_drop_lock()) releasing lock with cookie 0x37cab3886ce5d2c2 from it ffff8800a7343cd8 00010000:00010000:2.0:1734016148.654969:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1734016148.654975:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8800b5805800 lock: ffff8800a6c218c0/0x37cab3886ce5d2c2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb46129d8d3178702 expref: -99 pid: 10366 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1734016148.655012:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016148.655031:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016148.655068:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.655072:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e26f0, osc lock ffff8800aafcf1d0, flags 0x400001002 00000008:00010000:2.0:1734016148.655081:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.655082:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2730, osc lock ffff8800aafce850, flags 0x400001002 00010000:00010000:2.0:1734016148.655129:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d5d2 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.655136:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbd200/0x37cab3886ce5d5d2 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.655350:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d5d9 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.655359:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf840/0x37cab3886ce5d5d9 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.655445:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332569600:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016148.655588:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016148.655595:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332569472:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016148.661085:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016148.662148:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016148.662175:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334660736 00000100:00100000:2.0:1734016148.662179:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334660736 00000100:00100000:2.0:1734016148.662205:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14877 00000100:00100000:2.0:1734016148.662213:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73cb480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334660736:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016148.662240:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.662246:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.662267:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 6, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016148.662268:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 344 00000020:00010000:2.0:1734016148.662271:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016148.662272:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 344 00000100:00100000:2.0:1734016148.662292:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73cb480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334660736:12345-192.168.201.152@tcp:106: Request processed in 80us (146us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016148.662298:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14877 00000100:00100000:2.0:1734016148.680498:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016148.680890:0:1963:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016148.680925:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334660864 00000100:00100000:2.0:1734016148.680937:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334660864 00000100:00100000:2.0:1734016148.680990:0:4729:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334660992 00000100:00100000:2.0:1734016148.680992:0:4729:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334660992 00000100:00100000:2.0:1734016148.681009:0:4729:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14878 00000100:00100000:2.0:1734016148.681020:0:4729:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73c8700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334660864:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016148.681043:0:4729:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.681054:0:4729:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.681091:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 6, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016148.681094:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 176 00000020:00010000:2.0:1734016148.681098:0:4729:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 2, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016148.681099:0:4729:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 176 00000100:00100000:2.0:1734016148.681875:0:4729:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73c8700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13917:x1818247334660864:12345-192.168.201.152@tcp:106: Request processed in 855us (1395us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016148.681885:0:4729:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14878 00000080:00200000:2.0:1734016148.690822:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 520 00000020:00010000:2.0:1734016148.690842:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.690847:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.690853:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676cf0 (ffff88012db7c098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.690862:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a8676d30 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.690866:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.691298:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676cf0 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.691306:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a8676d30 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.691328:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016148.691347:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9625452 00000080:00200008:2.0:1734016148.691417:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 8125202, type: SEEK_SET 00000080:00200008:2.0:1734016148.691429:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 8125202, type: SEEK_SET, rc = 8125202 00000080:00200008:2.0:1734016148.691443:0:10366:0:(file.c:2466:do_file_write_iter()) START file f16b.sanityn:[0x200000401:0xc:0x0], ppos: 8125202, count: 112 00000080:00200000:2.0:1734016148.691460:0:10366:0:(rw26.c:819:ll_write_begin()) Writing 1983 of 2834 to 112 bytes 00000080:00200000:2.0:1734016148.691468:0:10366:0:(file.c:2441:ll_do_tiny_write()) result: 0, original count 112 00000080:00200000:2.0:1734016148.691483:0:10366:0:(file.c:1826:ll_file_io_generic()) f16b.sanityn: write ppos: 8125202, bytes: 112 00000080:00200000:2.0:1734016148.691488:0:10366:0:(file.c:1603:ll_io_set_mirror()) f16b.sanityn: desiginated mirror: 0 00000020:00200000:2.0:1734016148.691496:0:10366:0:(cl_io.c:195:cl_io_rw_init()) header@ffff8800b62c6a28[0x4, 6, [0x200000401:0xc:0x0] hash] 00000020:00200000:2.0:1734016148.691498:0:10366:0:(cl_io.c:195:cl_io_rw_init()) io range: 2 [8125202, 8125314) 0 0 00000080:00200000:2.0:1734016148.691502:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016148.691519:0:10366:0:(file.c:1893:ll_file_io_generic()) Range lock [1983, 1983] 00020000:00200000:2.0:1734016148.691527:0:10366:0:(lov_io.c:1021:lov_io_rw_iter_init()) stripe: 1 chunk: [8125202, 8125314) 8125314, 112 00020000:00200000:2.0:1734016148.691532:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016148.691556:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1983, 1983] 00000008:00010000:2.0:1734016148.691586:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016148.691593:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8c30, osc lock ffff8800aafcf430, flags 0x0 00010000:00010000:2.0:1734016148.691627:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.691653:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (3928064 3932159) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.691669:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.691677:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.691683:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016148.691692:0:10366:0:(vvp_io.c:1326:vvp_io_write_start()) f16b.sanityn: write [8125202, 8125314) 00000080:00200000:2.0:1734016148.691701:0:10366:0:(rw26.c:819:ll_write_begin()) Writing 1983 of 2834 to 112 bytes 00010000:00010000:2.0:1734016148.691776:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.691783:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 4/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.691790:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 7/1,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.691795:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 7/1,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.691804:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 6/1,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.697245:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 6/1,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.697279:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.789811:0:1971:0:(osc_cache.c:824:osc_extent_finish()) ### extent: ffff8800aa7e2240 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.790301:0:1971:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2240 ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.790336:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332570112:192.168.201.152@tcp:3:fsx.0 00000100:00100000:2.0:1734016148.816956:0:9532:0:(service.c:171:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1734016148.817008:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334661120 00000100:00100000:2.0:1734016148.817025:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334661120 00000100:00100000:2.0:1734016148.819137:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14880 00000100:00100000:2.0:1734016148.819159:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a7039c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334661120:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016148.819344:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.819362:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.819414:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1030, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016148.819420:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 344 00000020:00010000:2.0:1734016148.819424:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016148.819425:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 344 00000100:00100000:2.0:1734016148.829733:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a7039c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13917:x1818247334661120:12345-192.168.201.152@tcp:106: Request processed in 10539us (23575us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016148.829763:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14880 00000080:00200000:2.0:1734016148.840313:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 520 00000020:00010000:2.0:1734016148.840322:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.840325:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.840330:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86764b0 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.840334:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a86764f0 (ffff88012db7c878/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.840336:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a70dcff8 (ffff8800a72d97e8/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.840338:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86764b0 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.840343:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a86764f0 (ffff8800a72d97e8/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.840359:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016148.840415:0:10366:0:(file.c:5051:ll_file_seek()) START file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1734016148.840424:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016148.840434:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016148.840448:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.840451:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f30, osc lock ffff8800aafce130, flags 0x400001002 00000008:00010000:2.0:1734016148.840457:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.840459:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f70, osc lock ffff8800aafce000, flags 0x400001002 00010000:00010000:2.0:1734016148.840492:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d5fc lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.840505:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbf600/0x37cab3886ce5d5fc lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.840600:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d603 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.840607:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d603 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.840655:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332570624:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016148.840733:0:1970:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1734016148.840739:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332570496:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016148.847559:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016148.847626:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016148.847670:0:9532:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334661504 00000100:00100000:2.0:1734016148.847684:0:9532:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334661504 00000100:00100000:2.0:1734016148.847733:0:9532:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14881 00000100:00100000:2.0:1734016148.847741:0:9532:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73cb480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334661376:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016148.847768:0:9532:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.847781:0:9532:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.847815:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1030, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016148.847818:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 176 00000020:00010000:2.0:1734016148.847822:0:9532:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 3, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016148.847823:0:9532:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 176 00000100:00100000:2.0:1734016148.847882:0:9532:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73cb480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:7692:x1818247334661376:12345-192.168.201.152@tcp:106: Request processed in 140us (335us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016148.847889:0:9532:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14881 00000080:00200000:2.0:1734016148.861593:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 520 00000020:00010000:2.0:1734016148.861601:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1ff0 (ffff88012db7c878/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.861605:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1ff0 (ffff88012db7c878/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.861610:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f30 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.861614:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f70 (ffff88012db7c098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.861617:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1ff0 (ffff88012db7c878/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.861619:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f30 (ffff88012db7c878/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.861626:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f70 (ffff88012db7c878/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.861644:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1734016148.861664:0:10366:0:(file.c:5083:ll_file_seek()) COMPLETED file f16b.sanityn:[0x200000401:0xc:0x0], offset: 0, type: (null), rc = 9625452 00000080:00a00000:2.0:1734016148.861767:0:10366:0:(llite_mmap.c:611:ll_file_mmap()) VFS_Op: fid=[0x200000401:0xc:0x0] vma=ffff8800a9182798 start=0x7fef68fb7000 end=0x7fef68fb9000 vm_flags=0xfb 00000080:00200000:2.0:1734016148.861780:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1734016148.861791:0:10366:0:(glimpse.c:90:cl_glimpse_lock()) Glimpsing inode [0x200000401:0xc:0x0] 00000008:00010000:2.0:1734016148.861807:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.861810:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f30, osc lock ffff8800aafce000, flags 0x400001002 00000008:00010000:2.0:1734016148.861816:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1734016148.861817:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800aa7e2f70, osc lock ffff8800aafce130, flags 0x400001002 00010000:00010000:2.0:1734016148.861842:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d60a lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.861851:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff8800b5805800 lock: ffff88012dbbed00/0x37cab3886ce5d60a lrc: 3/1,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.861943:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d611 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.861951:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbf3c0/0x37cab3886ce5d611 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.862008:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332570752:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016148.862113:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016148.862119:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332570880:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016148.874309:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016148.874486:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000100:00100000:2.0:1734016148.874568:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334661632 00000100:00100000:2.0:1734016148.874591:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334661632 00000100:00100000:2.0:1734016148.874656:0:4728:0:(service.c:2100:ptlrpc_server_handle_req_in()) unwrap req x1818247334661760 00000100:00100000:2.0:1734016148.874659:0:4728:0:(service.c:2165:ptlrpc_server_handle_req_in()) got req x1818247334661760 00000100:00100000:2.0:1734016148.874674:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14883 00000100:00100000:2.0:1734016148.874681:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73c9180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334661632:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016148.874707:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.874725:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.874785:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1030, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016148.874789:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 344 00000020:00010000:2.0:1734016148.874793:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6af0[0x4, 2, [0x280000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016148.874794:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 5431148 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 344 00000100:00100000:2.0:1734016148.874870:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73c9180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13917:x1818247334661632:12345-192.168.201.152@tcp:106: Request processed in 188us (590us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016148.874877:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14883 00000100:00100000:2.0:1734016148.874899:0:4728:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.152@tcp, seq: 14884 00000100:00100000:2.0:1734016148.874904:0:4728:0:(service.c:2344:ptlrpc_server_handle_request()) Handling RPC req@ffff8800a73cb800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7694:x1818247334661760:12345-192.168.201.152@tcp:106: 00010000:00010000:2.0:1734016148.874912:0:4728:0:(ldlm_lockd.c:2536:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.874920:0:4728:0:(ldlm_lockd.c:2135:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff8800b3cc4800 lock: ffff88012dbbcd80/0x37cab3886ce5d5cb lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4034560->4038655) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a0b expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.874932:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6a28[0x4, 1030, [0x200000401:0xc:0x0] hash] 00000020:00010000:2.0:1734016148.874933:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 176 00000020:00010000:2.0:1734016148.874937:0:4728:0:(cl_object.c:300:cl_object_glimpse()) header@ffff8800b62c6bb8[0x4, 3, [0x240000400:0x7:0x0] hash] 00000020:00010000:2.0:1734016148.874938:0:4728:0:(cl_object.c:300:cl_object_glimpse()) size: 4194304 mtime: 1734016148 atime: 1734016148 ctime: 1734016148 blocks: 176 00000100:00100000:2.0:1734016148.874955:0:4728:0:(service.c:2396:ptlrpc_server_handle_request()) Handled RPC req@ffff8800a73cb800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:7694:x1818247334661760:12345-192.168.201.152@tcp:106: Request processed in 49us (469us total) trans 0 rc 0/0 00000100:00100000:2.0:1734016148.874961:0:4728:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.152@tcp, seq: 14884 00000080:00200000:2.0:1734016148.884802:0:10366:0:(file.c:1530:ll_merge_attr_nolock()) [0x200000401:0xc:0x0] updating i_size 9625452 i_blocks 520 00000020:00010000:2.0:1734016148.884810:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8801309c1ff0 (ffff88012db7c878/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.884813:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8801309c1ff0 (ffff88012db7c878/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.884818:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f30 (ffff88012db7c098/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.884823:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800aa7e2f70 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.884825:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8801309c1ff0 (ffff88012db7c878/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.884827:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f30 (ffff88012db7c878/1) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.884834:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800aa7e2f70 (ffff88012db7c878/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.884848:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016148.884945:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 1413 00000080:00200000:2.0:1734016148.884958:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016148.884977:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016148.884994:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1413, 1413] 00000008:00010000:2.0:1734016148.885014:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016148.885019:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8830, osc lock ffff8800aafce130, flags 0x0 00010000:00010000:2.0:1734016148.885050:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.885058:0:10366:0:(ldlm_request.c:1018:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [1593344->1597439] (req 1593344->1597439) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.885111:0:10366:0:(ldlm_request.c:1105:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 3/0,1 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [1593344->1597439] (req 1593344->1597439) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1734016148.885144:0:10366:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job fsx:5762f609-0e87-4e8c-9bd6-d528092c9d9c:10366:1818247332571008:192.168.201.152@tcp:101:fsx.0 00000100:00100000:2.0:1734016148.885269:0:10366:0:(client.c:2562:ptlrpc_set_wait()) set ffff8800aa7e2840 going to sleep for 11 seconds 00000100:00100000:2.0:1734016148.898126:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000080:00200000:2.0:1734016148.915026:0:4736:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000100:00100000:2.0:1734016148.938991:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800b341c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332571264:192.168.201.152@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1734016148.954851:0:1962:0:(events.c:359:request_in_callback()) peer: 12345-192.168.201.152@tcp (source: 12345-192.168.201.152@tcp) 00000020:00010000:2.0:1734016148.966994:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724e998 (ffff88012db7c878/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.966999:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724e998 (ffff88012db7c878/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.967003:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8830 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016148.967018:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.967031:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.967045:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724e998 (ffff88012db7c878/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.967046:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8830 (ffff88012db7c878/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.967066:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016148.967084:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016148.967095:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016148.967105:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1413, 1413] 00000008:00010000:2.0:1734016148.967114:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016148.967117:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8830, osc lock ffff8800aafce130, flags 0x0 00010000:00010000:2.0:1734016148.967131:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.967144:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1593344 1597439) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 2/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.967152:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.967155:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 6/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.967160:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.967187:0:10366:0:(osc_cache.c:783:osc_extent_find()) ### extent: ffff8800aa7e2600 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 4/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016148.967201:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af698248, count 1, dirtied 1 00000020:00010000:2.0:1734016148.967210:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724e998 (ffff88012db7c878/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.967211:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724e998 (ffff88012db7c878/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.967212:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8830 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016148.967241:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.967246:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.967250:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724e998 (ffff88012db7c878/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.967251:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8830 (ffff88012db7c878/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.967255:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016148.967290:0:10366:0:(rw.c:1991:ll_readpage()) fast read pgno: 1414 00000080:00200000:2.0:1734016148.967295:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016148.967298:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016148.967301:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1414, 1414] 00000008:00010000:2.0:1734016148.967304:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016148.967306:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8830, osc lock ffff8800aafce130, flags 0x0 00010000:00010000:2.0:1734016148.967309:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.967314:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1597440 1601535) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.967321:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.967325:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.967328:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.972064:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724e998 (ffff88012db7c878/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.972070:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724e998 (ffff88012db7c878/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.972075:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8830 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016148.972084:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.972094:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.972113:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724e998 (ffff88012db7c878/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.972116:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8830 (ffff88012db7c878/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.972142:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016148.972164:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016148.972179:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1734016148.972194:0:10366:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [1414, 1414] 00000008:00010000:2.0:1734016148.972208:0:10366:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1734016148.972213:0:10366:0:(osc_lock.c:1238:osc_lock_init()) ### lock ffff8800b6ef8830, osc lock ffff8800aafce130, flags 0x0 00010000:00010000:2.0:1734016148.972259:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.972275:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1597440 1601535) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 3/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.972289:0:10366:0:(ldlm_lock.c:765:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.972295:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 7/0,2 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.972303:0:10366:0:(ldlm_lock.c:907:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 6/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.972366:0:10366:0:(osc_cache.c:543:osc_extent_merge()) ### extent: ffff8800aa7e2900 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.972375:0:10366:0:(osc_cache.c:484:__osc_extent_remove()) ### extent: ffff8800aa7e2900 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.972383:0:10366:0:(osc_cache.c:560:osc_extent_merge()) ### extent: ffff8800aa7e2600 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.972397:0:10366:0:(osc_cache.c:771:osc_extent_find()) ### extent: ffff8800aa7e2600 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016148.972426:0:10366:0:(vvp_io.c:1133:vvp_set_batch_dirty()) mapping ffff8800af698248, count 1, dirtied 1 00000020:00010000:2.0:1734016148.972441:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) release lock: ffff8800a724e998 (ffff88012db7c878/0) at cl_lock_release():233 00000020:00010000:2.0:1734016148.972444:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800a724e998 (ffff88012db7c878/0) at cl_lock_cancel():150 00000020:00010000:2.0:1734016148.972446:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: ffff8800b6ef8830 (ffff88012db7c2d8/1) at cl_lock_cancel():150 00010000:00010000:2.0:1734016148.972450:0:10366:0:(ldlm_lock.c:819:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 5/0,1 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x810020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.972458:0:10366:0:(ldlm_lock.c:900:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 4/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000020:00010000:2.0:1734016148.972466:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800a724e998 (ffff88012db7c878/0) at cl_lock_fini():85 00000020:00010000:2.0:1734016148.972468:0:10366:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: ffff8800b6ef8830 (ffff88012db7c878/1) at cl_lock_fini():85 00000080:00200000:2.0:1734016148.972485:0:10366:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1734016148.972526:0:10366:0:(file.c:5185:ll_fsync()) VFS Op:inode=[0x200000401:0xc:0x0](ffff8800af6980c8), start 5787648, end 5795839, datasync 1 00000080:00200000:2.0:1734016148.972537:0:10366:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1734016148.972540:0:10366:0:(lov_io.c:849:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1734016148.972554:0:10366:0:(osc_cache.c:1823:try_to_add_extent_for_io()) ### extent: ffff8800aa7e2600 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1734016148.972568:0:10366:0:(osc_cache.c:1066:osc_extent_make_ready()) ### extent: ffff8800aa7e2600 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016148.972683:0:10366:0:(ldlm_lock.c:1499:ldlm_lock_match_with_skip()) ### matched (1593344 1597439) ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800020000020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000008:00100000:2.0:1734016148.972799:0:10366:0:(osc_request.c:2005:osc_brw_prep_request()) brw rpc ffff8800a73cbb80 - object 0x240000400:7 offset 1593344<>1601536 00000100:00100000:2.0:1734016148.972854:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1734016148.972875:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332571904:192.168.201.152@tcp:4:fsx.0 00010000:00010000:2.0:1734016149.009924:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016152.014363:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1734016152.278455:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800b341c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:356ef100-c450-4ad1-ae36-19e8c4618593:1971:1818247332572032:192.168.201.152@tcp:400:kworker.0 00000100:00100000:2.0:1734016152.278477:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800b341c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332572288:192.168.201.152@tcp:400:kworker.0 00000100:00100000:2.0:1734016152.278487:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800b341df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332572544:192.168.201.152@tcp:400:kworker.0 00000100:00100000:2.0:1734016152.278496:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800b341e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332572800:192.168.201.152@tcp:400:kworker.0 00010000:00010000:2.0:1734016155.020548:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1734016157.286523:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800b341f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:356ef100-c450-4ad1-ae36-19e8c4618593:1970:1818247332572928:192.168.201.152@tcp:400:kworker.0 00000100:00100000:2.0:1734016157.286572:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800b341ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332573184:192.168.201.152@tcp:400:kworker.0 00000100:00100000:2.0:1734016157.286591:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800b341ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332573440:192.168.201.152@tcp:400:kworker.0 00000100:00100000:2.0:1734016157.286608:0:1970:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800b341d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1970:1818247332573696:192.168.201.152@tcp:400:kworker.0 00010000:00010000:2.0:1734016158.026527:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016161.032417:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1734016162.294474:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800b341d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:356ef100-c450-4ad1-ae36-19e8c4618593:1971:1818247332573824:192.168.201.152@tcp:400:kworker.0 00000100:00100000:2.0:1734016162.294534:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800b341ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332574080:192.168.201.152@tcp:400:kworker.0 00000100:00100000:2.0:1734016162.294558:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332574336:192.168.201.152@tcp:400:kworker.0 00000100:00100000:2.0:1734016162.294573:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332574592:192.168.201.152@tcp:400:kworker.0 00010000:00010000:2.0:1734016164.038562:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1734016164.294472:0:1971:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016148/real 1734016148] req@ffff8800a73cbb80 x1818247332571904/t0(0) o4->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:6/4 lens 8680/448 e 0 to 1 dl 1734016164 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'fsx.0' uid:0 gid:0 00000100:00080000:2.0:1734016164.309086:0:1971:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1734016164.309094:0:1971:0:(import.c:200:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff8800b5805800: Connection to lustre-OST0000 (at 192.168.201.152@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1734016164.317045:0:1971:0:(import.c:432:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 00010000:00010000:2.0:1734016167.044524:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1734016168.317558:0:1971:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016152/real 1734016152] req@ffff8800b341c380 x1818247332572032/t0(0) o400->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 224/224 e 0 to 1 dl 1734016168 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734016168.317585:0:1971:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800b341c380 x1818247332572032/t0(0) o400->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 224/224 e 0 to 1 dl 1734016168 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734016168.317595:0:1971:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from FULL to DISCONN 00000100:02020000:2.0:1734016168.317599:0:1971:0:(import.c:204:ptlrpc_set_import_discon()) MGC192.168.201.152@tcp: Connection to MGS (at 192.168.201.152@tcp) was lost; in progress operations using this service will fail 00000100:00080000:2.0:1734016168.324094:0:1971:0:(import.c:210:ptlrpc_set_import_discon()) import MGS@192.168.201.152@tcp for MGC192.168.201.152@tcp not replayable, auto-deactivating 00000100:00080000:2.0:1734016168.324098:0:1971:0:(import.c:159:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:2.0:1734016168.324106:0:1971:0:(client.c:3445:ptlrpc_abort_inflight()) @@@ inflight req@ffff8800b341c380 x1818247332572032/t0(0) o400->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 224/224 e 0 to 1 dl 1734016168 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734016168.324118:0:1971:0:(client.c:3445:ptlrpc_abort_inflight()) @@@ inflight req@ffff8800b341f480 x1818247332572928/t0(0) o400->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 224/224 e 0 to 0 dl 1734016173 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734016168.324127:0:1971:0:(client.c:3445:ptlrpc_abort_inflight()) @@@ inflight req@ffff8800b341d180 x1818247332573824/t0(0) o400->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 224/224 e 0 to 0 dl 1734016178 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734016168.324136:0:1971:0:(client.c:3445:ptlrpc_abort_inflight()) @@@ inflight req@ffff8800a705d880 x1818247332574720/t0(0) o400->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 224/224 e 0 to 0 dl 1734016180 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 10000000:01000000:2.0:1734016168.324182:0:1971:0:(mgc_request.c:1140:mgc_import_event()) import event 0x808001 00000100:00080000:2.0:1734016168.324185:0:1971:0:(pinger.c:217:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:2.0:1734016168.324187:0:1971:0:(mgc_request.c:1140:mgc_import_event()) import event 0x808002 00000100:00080000:2.0:1734016168.324189:0:1971:0:(import.c:432:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00000400:2.0:1734016168.324208:0:1971:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016152/real 1734016152] req@ffff8800b341c700 x1818247332572288/t0(0) o400->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016168 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734016168.324248:0:1971:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800b341c700 x1818247332572288/t0(0) o400->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016168 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734016168.324255:0:1971:0:(import.c:232:ptlrpc_set_import_discon()) osc: import ffff8800aa6a0800 already not connected (conn 3, was 2): CONNECTING 00000100:00000400:2.0:1734016168.324262:0:1971:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016152/real 1734016152] req@ffff8800b341df80 x1818247332572544/t0(0) o400->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016168 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734016168.324273:0:1971:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800b341df80 x1818247332572544/t0(0) o400->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016168 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734016168.324281:0:1971:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1734016168.324284:0:1971:0:(import.c:200:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8800b3cc4800: Connection to lustre-MDT0000 (at 192.168.201.152@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1734016168.324288:0:1971:0:(import.c:432:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00000400:2.0:1734016168.324295:0:1971:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016152/real 1734016152] req@ffff8800b341e300 x1818247332572800/t0(0) o400->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016168 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734016168.324307:0:1971:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800b341e300 x1818247332572800/t0(0) o400->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016168 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734016168.324314:0:1971:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1734016168.324318:0:1971:0:(import.c:200:ptlrpc_set_import_discon()) lustre-OST0001-osc-ffff8800b3cc4800: Connection to lustre-OST0001 (at 192.168.201.152@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1734016168.324321:0:1971:0:(import.c:432:ptlrpc_pinger_force()) lustre-OST0001_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1734016168.324344:0:1971:0:(client.c:1722:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8800a73cdc00 x1818247332575872/t0(0) o400->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl Rpc:NWQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734016168.324357:0:1971:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332576128:192.168.201.152@tcp:400:kworker.0 00000100:00100000:2.0:1734016168.324416:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:356ef100-c450-4ad1-ae36-19e8c4618593:1971:1818247332573824:192.168.201.152@tcp:400:kworker.0 00000100:00100000:2.0:1734016168.324424:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:356ef100-c450-4ad1-ae36-19e8c4618593:1971:1818247332572032:192.168.201.152@tcp:400:kworker.0 00000100:00100000:2.0:1734016168.324431:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332572544:192.168.201.152@tcp:400:kworker.0 00000100:00100000:2.0:1734016168.324436:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1971:1818247332572800:192.168.201.152@tcp:400:kworker.0 00000100:00100000:2.0:1734016168.324465:0:1971:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:2.0:1734016168.324471:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:356ef100-c450-4ad1-ae36-19e8c4618593:0:1818247332575616:192.168.201.152@tcp:400:kworker.0 00000100:00100000:2.0:1734016168.324485:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:0:1818247332576000:192.168.201.152@tcp:400:kworker.0 00000100:00100000:2.0:1734016168.324498:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f1fa65c5-ac21-42fc-9fc8-6a110334833e:0:1818247332576256:192.168.201.152@tcp:400:kworker.0 00000100:00080000:2.0:1734016168.324654:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016168.324659:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016168.324663:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016168.324666:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734016168.324668:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016168.324674:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 369 00000100:00080000:2.0:1734016168.324684:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016168.324691:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016168.324710:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247332576896/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016169.326315:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016169.326324:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016169.326328:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016169.326332:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016169.326334:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016169.326338:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016169.326340:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016169.326343:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016169.326345:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016169.326348:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016169.326351:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016169.326353:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016169.326356:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016169.326359:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1734016170.050512:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016173.056494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1734016173.324388:0:1971:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016157/real 1734016157] req@ffff8800b341e680 x1818247332573056/t0(0) o400->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016173 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734016173.324421:0:1971:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800b341e680 x1818247332573056/t0(0) o400->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016173 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734016173.324431:0:1971:0:(import.c:232:ptlrpc_set_import_discon()) mdc: import ffff8800a9471000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:2.0:1734016173.324438:0:1971:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016157/real 1734016157] req@ffff8800b341dc00 x1818247332573312/t0(0) o400->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016173 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734016173.324450:0:1971:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800b341dc00 x1818247332573312/t0(0) o400->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016173 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734016173.324458:0:1971:0:(import.c:232:ptlrpc_set_import_discon()) osc: import ffff8800aa6a2000 already not connected (conn 3, was 2): CONNECTING 00000100:00000400:2.0:1734016173.324464:0:1971:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016157/real 1734016157] req@ffff8800b341ed80 x1818247332573568/t0(0) o400->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016173 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734016173.324475:0:1971:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800b341ed80 x1818247332573568/t0(0) o400->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016173 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734016173.324481:0:1971:0:(import.c:232:ptlrpc_set_import_discon()) osc: import ffff88012a632800 already not connected (conn 3, was 2): CONNECTING 00000100:00100000:2.0:1734016173.324494:0:1971:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1971:1818247332573056:192.168.201.152@tcp:400:kworker.0 00000100:00080000:2.0:1734016173.334322:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016173.334329:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016173.334332:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016173.334336:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016173.334338:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016173.334341:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016173.334344:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016173.334347:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016173.334349:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016173.334352:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016173.334355:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016173.334357:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016173.334360:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016173.334363:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1734016176.062513:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734016178.324658:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016178.324667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734016178.324671:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016178.324678:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 435 00000100:00080000:2.0:1734016178.324689:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1734016178.324696:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734016178.324730:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247332577152/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016178.324766:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016178.324770:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734016178.324772:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016178.324777:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 435 00000100:00080000:2.0:1734016178.324780:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1734016178.324784:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016178.324793:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734016178.324805:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247332577280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016178.324815:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016178.324819:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016178.324822:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016178.324825:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734016178.324827:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016178.324830:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 435 00000100:00080000:2.0:1734016178.324833:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1734016178.324837:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016178.324842:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016178.324850:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247332577408/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016178.324858:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016178.324861:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734016178.324862:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016178.324866:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 435 00000100:00080000:2.0:1734016178.324869:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1734016178.324872:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016178.324876:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734016178.324883:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247332577536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016178.324892:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016178.324895:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734016178.324896:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016178.324900:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 435 00000100:00080000:2.0:1734016178.324903:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1734016178.324907:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016178.324911:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016178.324918:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca300 x1818247332577664/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016178.324927:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016178.324930:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734016178.324932:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016178.324935:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 435 00000100:00080000:2.0:1734016178.324938:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1734016178.324942:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016178.324946:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016178.324953:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247332577792/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00000400:2.0:1734016178.342400:0:1970:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016162/real 1734016162] req@ffff8800b341f800 x1818247332573952/t0(0) o400->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016178 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734016178.342424:0:1970:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800b341f800 x1818247332573952/t0(0) o400->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 224/224 e 0 to 1 dl 1734016178 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734016178.342434:0:1970:0:(import.c:232:ptlrpc_set_import_discon()) mdc: import ffff8800a9471000 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:2.0:1734016178.342441:0:1970:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016162/real 1734016162] req@ffff8800a73cd880 x1818247332574208/t0(0) o400->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016178 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734016178.342453:0:1970:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a73cd880 x1818247332574208/t0(0) o400->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016178 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734016178.342459:0:1970:0:(import.c:232:ptlrpc_set_import_discon()) osc: import ffff8800aa6a2000 already not connected (conn 4, was 2): CONNECTING 00000100:00000400:2.0:1734016178.342465:0:1970:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734016162/real 1734016162] req@ffff8800a73cfb80 x1818247332574464/t0(0) o400->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016178 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734016178.342476:0:1970:0:(client.c:2407:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8800a73cfb80 x1818247332574464/t0(0) o400->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 224/224 e 0 to 1 dl 1734016178 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734016178.342483:0:1970:0:(import.c:232:ptlrpc_set_import_discon()) osc: import ffff88012a632800 already not connected (conn 4, was 2): CONNECTING 00000100:00100000:2.0:1734016178.342493:0:1970:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800b341f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1970:1818247332573952:192.168.201.152@tcp:400:kworker.0 00010000:00010000:2.0:1734016179.068484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734016179.326430:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016179.326438:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016179.326442:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016179.326445:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016179.326448:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016179.326451:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016179.326453:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016179.326456:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016179.326459:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016179.326461:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016179.326464:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016179.326467:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016179.326470:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016179.326472:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1734016182.074469:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734016183.334322:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016183.334333:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016183.334337:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016183.334341:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016183.334343:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016183.334346:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016183.334349:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016183.334352:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016183.334354:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016183.334357:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016183.334360:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016183.334363:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016183.334365:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016183.334368:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1734016185.080504:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016188.086510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016191.092474:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734016193.324667:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016193.324675:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734016193.324679:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016193.324687:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 445 00000100:00080000:2.0:1734016193.324697:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1734016193.324705:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734016193.324766:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247332578048/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016193.324798:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016193.324802:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734016193.324804:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016193.324808:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 445 00000100:00080000:2.0:1734016193.324812:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1734016193.324816:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016193.324826:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734016193.324839:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247332578176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016193.324850:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016193.324854:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016193.324857:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016193.324859:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734016193.324861:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016193.324864:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 445 00000100:00080000:2.0:1734016193.324867:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1734016193.324871:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016193.324876:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016193.324884:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247332578304/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016193.324892:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016193.324895:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734016193.324897:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016193.324901:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 445 00000100:00080000:2.0:1734016193.324904:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1734016193.324907:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016193.324911:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734016193.324919:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247332578432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016193.324927:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016193.324930:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734016193.324932:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016193.324936:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 445 00000100:00080000:2.0:1734016193.324939:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1734016193.324942:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016193.324947:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016193.324953:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247332578560/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016193.324962:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016193.324964:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734016193.324966:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016193.324969:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 445 00000100:00080000:2.0:1734016193.324972:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1734016193.324975:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016193.324980:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016193.324987:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247332578688/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:2.0:1734016194.098502:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734016194.326405:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016194.326413:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016194.326417:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016194.326420:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016194.326423:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016194.326426:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016194.326429:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016194.326431:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016194.326434:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016194.326437:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016194.326439:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016194.326442:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016194.326444:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016194.326447:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1734016197.104478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734016198.334347:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016198.334356:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016198.334360:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016198.334364:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016198.334367:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016198.334370:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016198.334373:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016198.334375:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016198.334378:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016198.334381:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016198.334384:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016198.334387:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016198.334389:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016198.334392:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1734016200.110477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016203.116530:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734016203.350332:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016203.350341:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016203.350344:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016203.350348:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016203.350351:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016203.350354:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016203.350358:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016203.350361:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016203.350363:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016203.350367:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016203.350369:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016203.350372:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016203.350375:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016203.350378:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000800:00000100:2.0:1734016204.331396:0:1959:0:(socklnd_cb.c:2394:ksocknal_find_timed_out_conn()) A connection with 12345-192.168.201.152@tcp (192.168.201.152:988) timed out; the network or node may be down. 00010000:00010000:2.0:1734016206.122506:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734016208.358365:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016208.358378:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016208.358382:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016208.358387:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016208.358390:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016208.358394:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016208.358396:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016208.358399:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016208.358402:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016208.358405:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016208.358410:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016208.358413:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016208.358416:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016208.358420:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1734016209.128552:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734016209.324557:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016209.324566:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016209.324569:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016209.324573:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016209.324576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016209.324579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734016209.324583:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016209.324590:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 456 00000100:00080000:2.0:1734016209.324601:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1734016209.324608:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016209.324620:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734016209.324650:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247332578816/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016209.324683:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016209.324688:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016209.324691:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016209.324694:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016209.324697:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016209.324700:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1734016209.324703:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016209.324706:0:5:0:(pinger.c:272:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1734016212.134496:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1734016212.330550:0:1955:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.201.152:988 00000400:00000100:2.0:1734016212.330597:0:1955:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1734016212.330618:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00080000:2.0:1734016213.324688:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016213.324697:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734016213.324701:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016213.324709:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 460 00000100:00080000:2.0:1734016213.324719:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1734016213.324726:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734016213.324756:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247332578944/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016213.324793:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016213.324797:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734016213.324800:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016213.324804:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 460 00000100:00080000:2.0:1734016213.324807:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1734016213.324811:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016213.324822:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734016213.324832:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca300 x1818247332579072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016213.324843:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016213.324846:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734016213.324848:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016213.324851:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 476 00000100:00080000:2.0:1734016213.324854:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016213.324858:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016213.324862:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734016213.324870:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247332579200/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016213.324879:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016213.324882:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734016213.324883:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016213.324886:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 460 00000100:00080000:2.0:1734016213.324890:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1734016213.324893:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016213.324898:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016213.324905:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247332579328/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016213.324913:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016213.324916:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734016213.324918:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016213.324922:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 460 00000100:00080000:2.0:1734016213.324924:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1734016213.324928:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016213.324932:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734016213.324941:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247332579456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016213.324949:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016213.324952:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734016213.324953:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016213.324957:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 460 00000100:00080000:2.0:1734016213.324960:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1734016213.324964:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016213.324968:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016213.324975:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247332579584/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016213.324983:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016213.324986:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734016213.324987:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016213.324990:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 460 00000100:00080000:2.0:1734016213.324994:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1734016213.324997:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016213.325001:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016213.325008:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247332579712/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:2.0:1734016215.140507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1734016215.336559:0:1957:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.201.152:988 00000400:00000100:2.0:1734016215.336604:0:1957:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1734016215.336623:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016215.336657:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016215.336666:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016215.336672:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016215.336679:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016215.336686:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016215.336692:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00010000:00010000:2.0:1734016218.146533:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734016218.342485:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016218.342494:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734016218.342499:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016218.342509:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 480 00000100:00080000:2.0:1734016218.342518:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016218.342526:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734016218.342560:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247332579840/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016218.342594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016218.342598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734016218.342600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016218.342605:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 480 00000100:00080000:2.0:1734016218.342608:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016218.342611:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016218.342622:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734016218.342633:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247332579968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016218.342643:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016218.342647:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734016218.342649:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016218.342652:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 480 00000100:00080000:2.0:1734016218.342655:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016218.342658:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016218.342663:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734016218.342671:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247332580096/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016218.342679:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016218.342683:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734016218.342684:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016218.342687:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 480 00000100:00080000:2.0:1734016218.342690:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016218.342693:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016218.342697:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016218.342705:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247332580224/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016218.342714:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016218.342717:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734016218.342719:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016218.342722:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 480 00000100:00080000:2.0:1734016218.342724:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016218.342728:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016218.342732:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734016218.342740:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247332580352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016218.342748:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016218.342751:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734016218.342753:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016218.342757:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 480 00000100:00080000:2.0:1734016218.342769:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016218.342772:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016218.342777:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016218.342785:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247332580480/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016218.342793:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016218.342796:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734016218.342798:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016218.342801:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 480 00000100:00080000:2.0:1734016218.342803:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016218.342806:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016218.342811:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016218.342818:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247332580608/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:2.0:1734016221.152507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734016223.350361:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016223.350371:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734016223.350375:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016223.350385:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 485 00000100:00080000:2.0:1734016223.350395:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016223.350403:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734016223.350440:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca300 x1818247332580736/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016223.350474:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016223.350478:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734016223.350480:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016223.350485:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 485 00000100:00080000:2.0:1734016223.350488:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016223.350492:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016223.350502:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734016223.350512:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247332580864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016223.350522:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016223.350526:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734016223.350527:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016223.350531:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 485 00000100:00080000:2.0:1734016223.350534:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016223.350538:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016223.350543:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734016223.350553:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247332580992/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016223.350561:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016223.350564:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734016223.350566:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016223.350569:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 485 00000100:00080000:2.0:1734016223.350571:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016223.350575:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016223.350579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016223.350586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247332581120/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016223.350595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016223.350598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734016223.350600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016223.350604:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 485 00000100:00080000:2.0:1734016223.350607:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016223.350610:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016223.350614:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734016223.350622:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247332581248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016223.350630:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016223.350634:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734016223.350636:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016223.350640:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 485 00000100:00080000:2.0:1734016223.350643:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016223.350646:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016223.350650:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016223.350657:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247332581376/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016223.350665:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016223.350668:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734016223.350669:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016223.350673:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 485 00000100:00080000:2.0:1734016223.350675:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016223.350678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016223.350683:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016223.350690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247332581504/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:2.0:1734016224.158483:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016227.164500:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734016228.358344:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016228.358354:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734016228.358358:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016228.358369:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 490 00000100:00080000:2.0:1734016228.358379:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016228.358387:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734016228.358423:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247332581632/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016228.358457:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016228.358461:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734016228.358463:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016228.358468:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 490 00000100:00080000:2.0:1734016228.358471:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016228.358475:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016228.358485:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734016228.358495:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247332581760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016228.358505:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016228.358508:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734016228.358510:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016228.358514:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 490 00000100:00080000:2.0:1734016228.358517:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016228.358520:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016228.358524:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734016228.358533:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247332581888/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016228.358541:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016228.358544:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734016228.358546:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016228.358550:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 490 00000100:00080000:2.0:1734016228.358553:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016228.358556:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016228.358561:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016228.358569:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247332582016/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016228.358577:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016228.358581:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734016228.358582:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016228.358585:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 490 00000100:00080000:2.0:1734016228.358588:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016228.358591:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016228.358595:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734016228.358604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247332582144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016228.358613:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016228.358616:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734016228.358617:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016228.358621:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 490 00000100:00080000:2.0:1734016228.358624:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016228.358627:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016228.358632:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016228.358639:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247332582272/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016228.358647:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016228.358651:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734016228.358652:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016228.358656:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 490 00000100:00080000:2.0:1734016228.358658:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016228.358661:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016228.358665:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016228.358672:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247332582400/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:2.0:1734016230.170506:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016233.176486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734016233.366326:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016233.366335:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734016233.366339:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016233.366349:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 495 00000100:00080000:2.0:1734016233.366358:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016233.366366:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734016233.366402:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247332582528/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016233.366436:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016233.366441:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734016233.366443:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016233.366447:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 495 00000100:00080000:2.0:1734016233.366450:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016233.366455:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016233.366466:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734016233.366476:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247332582656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016233.366486:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016233.366490:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734016233.366491:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016233.366495:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 495 00000100:00080000:2.0:1734016233.366497:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016233.366501:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016233.366506:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734016233.366516:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247332582784/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016233.366525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016233.366528:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734016233.366529:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016233.366533:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 495 00000100:00080000:2.0:1734016233.366535:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016233.366538:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016233.366543:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016233.366550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca300 x1818247332582912/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016233.366559:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016233.366563:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734016233.366565:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016233.366568:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 495 00000100:00080000:2.0:1734016233.366571:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016233.366574:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016233.366578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734016233.366586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247332583040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016233.366594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016233.366597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734016233.366598:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016233.366602:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 495 00000100:00080000:2.0:1734016233.366605:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016233.366608:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016233.366612:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016233.366620:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247332583168/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016233.366628:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016233.366631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734016233.366632:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016233.366635:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 495 00000100:00080000:2.0:1734016233.366648:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016233.366651:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016233.366657:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016233.366664:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247332583296/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:2.0:1734016236.182504:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016239.188492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016242.194477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016245.200472:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1734016246.388501:0:1955:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.201.152:988 00000400:00000100:2.0:1734016246.388551:0:1955:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1734016246.388575:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016246.388620:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016246.388632:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016246.388644:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016246.388654:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016246.388664:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016246.388675:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00010000:00010000:2.0:1734016248.206480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016251.212433:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016254.218486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016257.224466:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016260.230509:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016263.236436:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016266.242486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1734016266.420535:0:1957:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:2.0:1734016266.420583:0:1957:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1734016266.420601:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016266.420637:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016266.420645:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016266.420652:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016266.420658:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016266.420664:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016266.420670:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00010000:00010000:2.0:1734016269.248526:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016272.254491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016275.260534:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016278.266490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016281.272464:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016284.278507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016287.284405:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016290.290512:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016293.296484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734016293.462432:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016293.462444:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734016293.462450:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016293.462463:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 556 00000100:00080000:2.0:1734016293.462476:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016293.462486:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734016293.462527:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247332593280/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016293.462564:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016293.462568:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734016293.462570:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016293.462575:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 556 00000100:00080000:2.0:1734016293.462578:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016293.462583:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016293.462593:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734016293.462604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247332593408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016293.462615:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016293.462618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734016293.462620:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016293.462623:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 556 00000100:00080000:2.0:1734016293.462626:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016293.462629:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016293.462634:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734016293.462642:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247332593536/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016293.462651:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016293.462653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734016293.462655:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016293.462658:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 556 00000100:00080000:2.0:1734016293.462661:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016293.462664:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016293.462669:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016293.462677:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247332593664/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016293.462686:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016293.462689:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734016293.462691:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016293.462695:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 556 00000100:00080000:2.0:1734016293.462698:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016293.462701:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016293.462705:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734016293.462714:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247332593792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016293.462723:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016293.462726:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734016293.462727:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016293.462731:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 556 00000100:00080000:2.0:1734016293.462734:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016293.462737:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016293.462741:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016293.462749:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247332593920/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734016293.462757:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734016293.462760:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734016293.462761:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734016293.462765:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 556 00000100:00080000:2.0:1734016293.462767:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734016293.462771:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734016293.462775:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734016293.462782:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247332594048/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:2.0:1734016296.302482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016299.308547:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016302.314363:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016305.320418:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016308.326500:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016311.332447:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016314.338461:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016317.344484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1734016496.788661:0:1957:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:2.0:1734016496.788739:0:1957:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1734016496.788770:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016496.788841:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016496.788854:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016496.788866:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016496.788878:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016496.788889:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016496.788899:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00080000:1.0:1734016553.878735:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016553.878741:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332640000:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734016553.878749:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332640128:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016553.878767:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016553.878774:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016553/real 1734016553] req@ffff8800a73cd500 x1818247332640128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016553.878785:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332640128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016553.878793:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016553.878795:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016553.878799:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332640128:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016553.878805:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332640256:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016553.878818:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016553.878824:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016553/real 1734016553] req@ffff8800a73ce680 x1818247332640256/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016553.878835:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332640256/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016553.878842:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016553.878844:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016553.878847:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332640256:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016553.878872:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332640384:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016553.878885:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016553.878892:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016553/real 1734016553] req@ffff8800a73cc380 x1818247332640384/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016553.878902:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332640384/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016553.878909:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016553.878911:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016553.878914:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332640384:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016553.878920:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332640512:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016553.878931:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016553.878937:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016553/real 1734016553] req@ffff8800a73cce00 x1818247332640512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016553.878947:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332640512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016553.878955:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016553.878957:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016553.878960:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332640512:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016553.878966:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332640640:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016553.878977:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016553.878984:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016553/real 1734016553] req@ffff8800a73cf100 x1818247332640640/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016553.878993:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332640640/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016553.879008:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016553.879011:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016553.879014:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332640640:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016553.879019:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332640768:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016553.879031:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016553.879037:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016553/real 1734016553] req@ffff8800a73cea00 x1818247332640768/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016553.879046:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332640768/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016553.879053:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016553.879055:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016553.879058:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332640768:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016555.820513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016558.826497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1734016558.886617:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332640896:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734016558.886713:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332641024:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016558.886742:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332641152:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016558.886758:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332641280:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016558.886773:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332641408:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016558.886787:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332641536:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016558.886805:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332641664:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016561.832543:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1734016561.892706:0:1958:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:2.0:1734016561.892784:0:1958:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1734016561.892812:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016561.892880:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016561.892895:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016561.892907:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016561.892918:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016561.892931:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734016561.892943:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00010000:00010000:1.0:1734016564.838495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016567.844448:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016570.850526:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016573.856480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016576.862505:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016579.868478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016582.874508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016585.880512:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016588.886513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016591.892511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016594.898507:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016597.904531:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016598.906571:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734016598.907111:0:1970:0:(client.c:1722:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8800a73cc000 x1818247332648192/t0(0) o103->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:17/18 lens 424/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 00010000:00010000:1.0:1734016600.910476:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016603.916525:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016606.922553:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016609.928482:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016612.934516:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016615.940523:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016618.946487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016621.952552:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016624.958493:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1734016626.996603:0:1956:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:1.0:1734016626.996661:0:1956:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1734016626.996685:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734016626.996726:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734016626.996735:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734016626.996744:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734016626.996753:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734016626.996759:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734016626.996765:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00010000:00010000:1.0:1734016627.964568:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016630.970548:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016633.976483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016636.982480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016639.988536:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016642.994543:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016646.000504:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016648.004492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016649.006455:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016651.010462:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016652.012530:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016654.016482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016655.018494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016657.022491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016658.024482:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016660.028514:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016661.030477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016663.034481:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016664.036525:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016666.040477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016667.042516:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016669.046493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016670.048514:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016672.052478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016673.054518:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016675.058528:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016676.060448:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016678.064557:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016679.066522:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016681.070482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016682.072526:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016684.076563:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016685.078459:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016687.082520:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016688.084520:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016690.088516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016691.090508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1734016692.100666:0:1955:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:1.0:1734016692.100729:0:1955:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1734016692.100756:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734016692.100799:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734016692.100811:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734016692.100823:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734016692.100834:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734016692.100847:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734016692.100858:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00010000:00010000:2.0:1734016693.094479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016694.096496:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016696.100475:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016697.102535:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016699.106535:0:5236:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 2 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1734016699.106552:0:5236:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x800028400020000 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1734016699.106556:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1734016699.106585:0:5236:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016699.106714:0:5236:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1734016699.106738:0:5236:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbe1c0/0x37cab3886ce5911b lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0xb46129d8d3173a25 expref: -99 pid: 9631 timeout: 0 lvb_type: 1 00000080:00200000:2.0:1734016699.106749:0:5236:0:(vvp_io.c:1866:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1734016699.106755:0:5236:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1734016699.106795:0:5236:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 1/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x804869400020020 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016699.106804:0:5236:0:(ldlm_request.c:1327:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbe1c0/0x37cab3886ce5911b lrc: 1/0,0 mode: --/PR res: [0x280000400:0x5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0xb46129d8d3173a25 expref: -99 pid: 9631 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016699.106810:0:5236:0:(ldlm_request.c:1369:_ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1734016699.106846:0:5236:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff8800a6c23840/0x37cab3886ce5d57e lrc: 0/0,0 mode: --/PW res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 258048->266239) gid 0 flags: 0x804869400020020 nid: local remote: 0xb46129d8d31789be expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016699.106867:0:5236:0:(ldlm_lock.c:195:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8800b3cc4800 lock: ffff88012dbbe1c0/0x37cab3886ce5911b lrc: 0/0,0 mode: --/PR res: [0x280000400:0x5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0xb46129d8d3173a25 expref: -99 pid: 9631 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1734016699.106879:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016699.106882:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016699.106884:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016701.110481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016702.112477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016704.116465:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016705.118542:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016707.122501:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016708.124515:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016709.126503:0:4736:0:(ldlm_lockd.c:2838:ldlm_bl_get_work()) Got 3 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1734016709.126518:0:4736:0:(ldlm_request.c:1280:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800028400020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1734016709.126533:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016710.128441:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016711.130529:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016712.132501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016713.134536:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016714.136490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016715.138516:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016716.140452:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016717.142506:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016718.144503:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016719.146511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016720.148477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016721.150487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016722.152515:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016723.154504:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016724.156507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016725.158503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016726.160473:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016727.162534:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016728.164493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016729.158671:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016729.158674:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016729.158679:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 991 00000100:00080000:3.0:1734016729.158682:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016729.158686:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016729.158692:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016729.158703:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332672128/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016729.158713:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016729.158717:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016729.158719:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016729.158723:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 991 00000100:00080000:3.0:1734016729.158727:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016729.158730:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016729.158735:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016729.158744:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332672256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016729.158752:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016729.158755:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016729.158757:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016729.158761:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 991 00000100:00080000:3.0:1734016729.158765:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016729.158769:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016729.158773:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016729.158781:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332672384/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016729.158791:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016729.158794:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016729.158796:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016729.158800:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 991 00000100:00080000:3.0:1734016729.158803:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016729.158806:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016729.158810:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016729.158821:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332672512/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00010000:00010000:1.0:1734016729.166544:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016730.168449:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016731.170536:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016732.172474:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016733.174516:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016734.166395:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016734.166406:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016734.166411:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016734.166425:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 996 00000100:00080000:3.0:1734016734.166438:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016734.166447:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016734.166496:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332672640/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016734.166537:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016734.166597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016734.166600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016734.166606:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 996 00000100:00080000:3.0:1734016734.166610:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016734.166615:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016734.166628:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016734.166644:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332672768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016734.166657:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016734.166660:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016734.166662:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016734.166667:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 996 00000100:00080000:3.0:1734016734.166670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016734.166673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016734.166678:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016734.166688:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332672896/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016734.166697:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016734.166700:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016734.166702:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016734.166705:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 996 00000100:00080000:3.0:1734016734.166709:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016734.166713:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016734.166718:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016734.166726:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332673024/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016734.166734:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016734.166738:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016734.166740:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016734.166745:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 996 00000100:00080000:3.0:1734016734.166750:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016734.166754:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016734.166759:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016734.166768:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332673152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016734.166777:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016734.166780:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016734.166781:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016734.166786:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 996 00000100:00080000:3.0:1734016734.166788:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016734.166792:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016734.166796:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016734.166805:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332673280/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016734.166813:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016734.166816:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016734.166818:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016734.166821:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 996 00000100:00080000:3.0:1734016734.166824:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016734.166828:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016734.166832:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016734.166842:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332673408/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016734.166981:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332673152:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016734.166997:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016734.167003:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016734/real 1734016734] req@ffff8800a73cc380 x1818247332673152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016734.167014:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332673152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016734.167021:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016734.167024:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016734.167027:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332673152:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016734.167032:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332673280:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016734.167045:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016734.167051:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016734/real 1734016734] req@ffff8800a73cce00 x1818247332673280/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016734.167061:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332673280/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016734.167069:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016734.167071:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016734.167074:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332673280:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016734.167093:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332673408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016734.167105:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016734.167111:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016734/real 1734016734] req@ffff8800a73cf100 x1818247332673408/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016734.167128:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332673408/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016734.167136:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016734.167138:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016734.167141:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332673408:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734016734.176441:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016735.178508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016736.180488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016737.182590:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016738.184534:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016739.174356:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016739.174367:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016739.174374:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016739.174388:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1001 00000100:00080000:3.0:1734016739.174400:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016739.174410:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016739.174500:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332673536/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016739.174538:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016739.174542:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016739.174545:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016739.174550:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1001 00000100:00080000:3.0:1734016739.174553:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016739.174558:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016739.174571:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016739.174584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332673664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016739.174595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016739.174598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016739.174600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016739.174604:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1001 00000100:00080000:3.0:1734016739.174607:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016739.174610:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016739.174617:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016739.174626:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332673792/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016739.174635:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016739.174639:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016739.174640:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016739.174644:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1001 00000100:00080000:3.0:1734016739.174646:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016739.174650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016739.174654:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016739.174663:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332673920/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016739.174671:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332673536:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016739.174672:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016739.174676:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016739.174677:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016739.174681:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1001 00000100:00080000:3.0:1734016739.174684:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016739.174687:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016739.174692:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016739.174701:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332674048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016739.174709:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016739.174713:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016739.174715:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016739.174720:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1001 00000100:00080000:3.0:1734016739.174724:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016739.174727:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016739.174732:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016739.174739:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332674176/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734016739.174743:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016739.174747:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016739.174751:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016739.174752:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016739.174756:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1001 00000100:00080000:3.0:1734016739.174760:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016739.174763:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734016739.174765:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016739/real 1734016739] req@ffff8800a73cd500 x1818247332673536/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734016739.174768:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016739.174776:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332674304/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016739.174785:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332673536/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016739.174798:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016739.174801:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016739.174806:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332673536:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016739.174815:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332673664:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016739.174833:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016739.174840:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016739/real 1734016739] req@ffff8800a73cf800 x1818247332673664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016739.174851:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332673664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016739.174859:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016739.174863:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016739.174866:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332673664:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016739.174873:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332673792:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016739.174888:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016739.174894:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016739/real 1734016739] req@ffff8800a73ced80 x1818247332673792/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016739.174904:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332673792/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016739.174912:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016739.174914:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016739.174918:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332673792:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016739.174944:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332673920:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016739.174956:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016739.174963:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016739/real 1734016739] req@ffff8800a73cdf80 x1818247332673920/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016739.174973:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332673920/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016739.174981:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016739.174984:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016739.174987:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332673920:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016739.174993:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332674048:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016739.175009:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016739.175015:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016739/real 1734016739] req@ffff8800a73cf480 x1818247332674048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016739.175024:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332674048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016739.175032:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016739.175034:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016739.175037:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332674048:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016739.175043:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332674176:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016739.175055:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016739.175061:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016739/real 1734016739] req@ffff8800a73cd880 x1818247332674176/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016739.175070:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332674176/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016739.175078:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016739.175081:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016739.175084:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332674176:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016739.175089:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332674304:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016739.175102:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016739.175107:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016739/real 1734016739] req@ffff8800a73cfb80 x1818247332674304/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016739.175124:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332674304/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016739.175132:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016739.175134:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016739.175137:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332674304:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016739.186542:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016740.188524:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016741.190527:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016742.192493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016743.194569:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016744.182409:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016744.182419:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016744.182425:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016744.182439:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1006 00000100:00080000:3.0:1734016744.182451:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016744.182461:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016744.182512:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332674432/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016744.182558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016744.182562:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016744.182564:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016744.182570:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1006 00000100:00080000:3.0:1734016744.182574:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016744.182578:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016744.182589:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016744.182603:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332674560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016744.182612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016744.182616:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016744.182617:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016744.182622:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1006 00000100:00080000:3.0:1734016744.182625:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016744.182628:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016744.182633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016744.182642:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332674688/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016744.182651:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016744.182654:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016744.182656:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016744.182659:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1006 00000100:00080000:3.0:1734016744.182662:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016744.182667:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016744.182672:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016744.182681:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332674816/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016744.182686:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332674432:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016744.182691:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016744.182694:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016744.182696:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016744.182700:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1006 00000100:00080000:3.0:1734016744.182702:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016744.182706:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016744.182711:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016744.182719:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332674944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016744.182727:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016744.182731:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016744.182733:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016744.182738:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1006 00000100:00080000:3.0:1734016744.182741:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016744.182744:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016744.182749:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016744.182757:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332675072/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734016744.182762:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016744.182765:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016744.182769:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016744.182771:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016744.182775:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1006 00000100:00080000:3.0:1734016744.182779:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016744.182782:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734016744.182785:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016744/real 1734016744] req@ffff8800a73cca80 x1818247332674432/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734016744.182787:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016744.182795:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332675200/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016744.182806:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332674432/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016744.182818:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016744.182822:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016744.182827:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332674432:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016744.182835:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332674560:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016744.182853:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016744.182859:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016744/real 1734016744] req@ffff8800a73cfb80 x1818247332674560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016744.182870:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332674560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016744.182878:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016744.182882:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016744.182886:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332674560:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016744.182893:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332674688:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016744.182907:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016744.182913:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016744/real 1734016744] req@ffff8800a73cd880 x1818247332674688/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016744.182923:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332674688/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016744.182931:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016744.182933:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016744.182937:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332674688:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016744.182962:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332674816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016744.182975:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016744.182981:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016744/real 1734016744] req@ffff8800a73cf480 x1818247332674816/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016744.182992:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332674816/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016744.183000:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016744.183002:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016744.183005:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332674816:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016744.183011:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332674944:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016744.183027:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016744.183033:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016744/real 1734016744] req@ffff8800a73cdf80 x1818247332674944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016744.183043:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332674944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016744.183050:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016744.183053:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016744.183056:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332674944:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016744.183061:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332675072:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016744.183074:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016744.183079:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016744/real 1734016744] req@ffff8800a73ced80 x1818247332675072/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016744.183089:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332675072/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016744.183097:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016744.183099:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016744.183102:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332675072:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016744.183107:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332675200:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016744.183120:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016744.183125:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016744/real 1734016744] req@ffff8800a73cf800 x1818247332675200/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016744.183141:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332675200/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016744.183148:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016744.183151:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016744.183153:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332675200:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734016744.196538:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016745.198551:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016746.200542:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016747.202558:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016748.204552:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00020000:2.0:1734016748.972408:0:10366:0:(osc_cache.c:908:osc_extent_wait()) extent ffff8800aa7e2600@{[389 -> 390/1023], [4|0|+|rpc|wiuY|ffff8800a86f29c0], [32768|2|+|+|ffff88012dbbd8c0|1024|ffff8800a975e660]} lustre-OST0000-osc-ffff8800b5805800: wait ext to 0 timedout, recovery in progress? 00000008:00020000:2.0:1734016748.983281:0:10366:0:(osc_cache.c:908:osc_extent_wait()) ### extent: ffff8800aa7e2600 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800029400020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00000100:00080000:3.0:1734016749.190389:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016749.190398:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016749.190402:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016749.190412:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1011 00000100:00080000:3.0:1734016749.190422:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016749.190430:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016749.190467:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332675328/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016749.190499:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016749.190503:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016749.190505:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016749.190510:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1011 00000100:00080000:3.0:1734016749.190513:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016749.190517:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016749.190526:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016749.190537:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332675456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016749.190548:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016749.190551:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016749.190552:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016749.190556:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1011 00000100:00080000:3.0:1734016749.190558:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016749.190562:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016749.190566:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016749.190574:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332675584/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016749.190583:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016749.190586:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016749.190587:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016749.190591:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1011 00000100:00080000:3.0:1734016749.190593:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016749.190596:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016749.190601:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016749.190603:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332675328:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734016749.190608:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332675712/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016749.190618:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016749.190621:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016749.190622:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016749.190626:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1011 00000100:00080000:3.0:1734016749.190628:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016749.190632:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016749.190636:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016749.190643:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332675840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016749.190652:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016749.190655:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016749.190656:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016749.190660:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1011 00000100:00080000:3.0:1734016749.190663:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016749.190666:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734016749.190669:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734016749.190670:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016749.190677:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332675968/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016749.190685:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016749.190688:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734016749.190690:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016749/real 1734016749] req@ffff8800a73cd500 x1818247332675328/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016749.190690:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016749.190693:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1011 00000100:00080000:3.0:1734016749.190696:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016749.190699:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016749.190703:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016749.190709:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332675328/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016749.190712:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332676096/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016749.190720:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016749.190723:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016749.190728:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332675328:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016749.190735:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332675456:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016749.190752:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016749.190759:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016749/real 1734016749] req@ffff8800a73cf100 x1818247332675456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016749.190769:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332675456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016749.190778:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016749.190781:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016749.190785:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332675456:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016749.190791:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332675584:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016749.190805:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016749.190811:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016749/real 1734016749] req@ffff8800a73cce00 x1818247332675584/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016749.190821:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332675584/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016749.190828:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016749.190831:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016749.190834:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332675584:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016749.190858:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332675712:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016749.190870:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016749.190876:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016749/real 1734016749] req@ffff8800a73cc380 x1818247332675712/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016749.190887:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332675712/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016749.190895:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016749.190897:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016749.190900:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332675712:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016749.190904:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332675840:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016749.190919:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016749.190925:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016749/real 1734016749] req@ffff8800a73ce680 x1818247332675840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016749.190936:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332675840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016749.190944:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016749.190946:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016749.190949:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332675840:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016749.190953:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332675968:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016749.190964:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016749.190970:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016749/real 1734016749] req@ffff8800a73cc700 x1818247332675968/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016749.190980:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332675968/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016749.190987:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016749.190989:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016749.190992:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332675968:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016749.190997:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332676096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016749.191008:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016749.191014:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016749/real 1734016749] req@ffff8800a73cea00 x1818247332676096/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016749.191031:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332676096/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016749.191038:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016749.191040:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016749.191043:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332676096:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016749.206504:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016750.208517:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016751.210486:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016752.212479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016753.214526:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016754.198389:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016754.198399:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016754.198403:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016754.198414:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1016 00000100:00080000:3.0:1734016754.198424:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016754.198432:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016754.198468:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332676224/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016754.198503:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016754.198507:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016754.198509:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016754.198514:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1016 00000100:00080000:3.0:1734016754.198517:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016754.198521:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016754.198531:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016754.198545:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332676352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016754.198555:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016754.198558:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016754.198560:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016754.198564:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1016 00000100:00080000:3.0:1734016754.198567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016754.198570:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016754.198575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016754.198583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332676480/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016754.198593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016754.198596:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016754.198597:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016754.198601:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1016 00000100:00080000:3.0:1734016754.198603:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016754.198606:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016754.198611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016754.198618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332676608/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016754.198627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016754.198631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016754.198632:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016754.198636:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1016 00000100:00080000:3.0:1734016754.198638:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016754.198641:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734016754.198643:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332676224:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734016754.198646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016754.198653:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332676736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016754.198662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016754.198665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016754.198667:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016754.198672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1016 00000100:00080000:3.0:1734016754.198675:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016754.198678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016754.198683:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016754.198690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332676864/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016754.198698:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016754.198701:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016754.198703:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016754.198706:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1016 00000100:00080000:3.0:1734016754.198708:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016754.198711:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016754.198716:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016754.198723:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332676992/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016754.198768:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332676352:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016754.198793:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332676480:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016754.198823:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332676608:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016754.198839:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332676736:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016754.198858:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332676864:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016754.198875:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332676992:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734016754.216477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016755.218523:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016756.220497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734016757.204646:0:1957:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734016757.204714:0:1957:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734016757.204739:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734016757.204784:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734016757.204793:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734016757.204801:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734016757.204808:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734016757.204817:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734016757.204825:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734016757.204836:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016754/real 1734016757] req@ffff8800a73ce680 x1818247332676608/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016757.204872:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332676608/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016757.204886:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016757.204891:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016757.204897:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332676608:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016757.204906:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016754/real 1734016757] req@ffff8800a73cc380 x1818247332676736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016757.204919:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332676736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016757.204928:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016757.204931:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016757.204935:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332676736:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016757.204942:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016754/real 1734016757] req@ffff8800a73cce00 x1818247332676864/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016757.204953:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332676864/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016757.204960:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016757.204974:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016757.204977:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332676864:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016757.204985:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016754/real 1734016757] req@ffff8800a73cf100 x1818247332676992/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016757.204997:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332676992/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016757.205004:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016757.205007:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016757.205010:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332676992:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016757.205017:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016754/real 1734016757] req@ffff8800a73cdc00 x1818247332676224/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016757.205029:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332676224/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016757.205036:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016757.205040:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016757.205043:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332676224:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734016757.205049:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016754/real 1734016757] req@ffff8800a73cea00 x1818247332676352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016757.205060:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332676352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016757.205067:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016757.205070:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016757.205074:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332676352:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016757.205083:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016754/real 1734016757] req@ffff8800a73cc700 x1818247332676480/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016757.205102:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332676480/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016757.205115:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016757.205119:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016757.205124:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332676480:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016757.222511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016758.224516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016759.206387:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016759.206399:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016759.206405:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016759.206418:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1021 00000100:00080000:3.0:1734016759.206430:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016759.206440:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016759.206487:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332677120/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016759.206536:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016759.206543:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016759.206546:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016759.206553:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1021 00000100:00080000:3.0:1734016759.206558:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016759.206564:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016759.206579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016759.206599:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332677248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016759.206614:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734016759.206616:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332677120:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016759.206619:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016759.206622:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016759.206628:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1021 00000100:00080000:3.0:1734016759.206632:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016759.206637:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016759.206645:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016759.206659:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332677376/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016759.206671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016759.206675:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016759.206677:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016759.206683:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1021 00000100:00080000:3.0:1734016759.206701:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734016759.206707:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016759.206708:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016759.206715:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016759.206728:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332677504/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016759.206737:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016759/real 1734016759] req@ffff8800a73cd500 x1818247332677120/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016759.206742:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016759.206748:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016759.206750:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016759.206756:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1021 00000100:00080000:3.0:1734016759.206761:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734016759.206764:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332677120/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016759.206766:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016759.206774:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1734016759.206779:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016759.206784:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734016759.206787:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332677632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016759.206791:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332677120:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016759.206801:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016759.206806:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016759.206808:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016759.206814:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1021 00000100:00080000:3.0:1734016759.206818:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016759.206823:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734016759.206825:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332677248:192.168.201.152@tcp:38:kworker.0 00000008:00100000:3.0:1734016759.206830:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016759.206843:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332677760/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734016759.206854:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016759.206858:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016759.206864:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734016759.206865:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016759/real 1734016759] req@ffff8800a73cf800 x1818247332677248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016759.206867:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016759.206873:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1021 00000100:00080000:3.0:1734016759.206877:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016759.206883:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016759.206890:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734016759.206896:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332677248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016759.206904:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332677888/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734016759.206909:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016759.206913:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016759.206919:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332677248:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016759.206927:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332677376:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016759.206956:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016759.206968:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016759/real 1734016759] req@ffff8800a73ced80 x1818247332677376/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016759.206986:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332677376/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016759.206999:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016759.207002:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016759.207006:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332677376:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016759.207013:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332677504:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016759.207034:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016759.207042:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016759/real 1734016759] req@ffff8800a73cdf80 x1818247332677504/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016759.207053:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332677504/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016759.207061:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016759.207064:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016759.207067:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332677504:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016759.207073:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332677632:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016759.207087:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016759.207093:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016759/real 1734016759] req@ffff8800a73cf480 x1818247332677632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016759.207103:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332677632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016759.207111:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016759.207155:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016759.207160:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332677632:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016759.207185:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332677760:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016759.207204:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016759.207211:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016759/real 1734016759] req@ffff8800a73cd880 x1818247332677760/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016759.207267:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332677760/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016759.207275:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016759.207278:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016759.207281:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332677760:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016759.207287:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332677888:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016759.207307:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016759.207314:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016759/real 1734016759] req@ffff8800a73cfb80 x1818247332677888/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016759.207325:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332677888/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016759.207333:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016759.207335:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016759.207338:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332677888:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016759.226502:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016760.228431:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016761.230476:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016762.232460:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016763.234510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016764.214350:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016764.214361:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016764.214366:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016764.214382:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1026 00000100:00080000:3.0:1734016764.214396:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016764.214406:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016764.214457:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332678016/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016764.214499:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016764.214506:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016764.214509:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016764.214517:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1026 00000100:00080000:3.0:1734016764.214521:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016764.214529:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016764.214545:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016764.214569:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332678144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016764.214578:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332678016:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016764.214586:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016764.214591:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016764.214593:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016764.214599:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1026 00000100:00080000:3.0:1734016764.214603:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016764.214608:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016764.214615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016764.214627:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332678272/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016764.214639:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016764.214643:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016764.214646:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016764.214651:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1026 00000100:00080000:3.0:1734016764.214667:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016764.214672:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734016764.214674:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734016764.214680:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016764.214694:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332678400/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016764.214705:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016764/real 1734016764] req@ffff8800a73cca80 x1818247332678016/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016764.214706:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016764.214711:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016764.214714:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016764.214720:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1026 00000100:00080000:3.0:1734016764.214725:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016764.214730:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734016764.214732:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332678016/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734016764.214737:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1734016764.214747:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1734016764.214751:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332678528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734016764.214752:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016764.214759:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332678016:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016764.214765:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016764.214769:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016764.214771:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016764.214777:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1026 00000100:00080000:3.0:1734016764.214781:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016764.214786:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734016764.214792:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332678144:192.168.201.152@tcp:38:kworker.0 00000008:00100000:3.0:1734016764.214792:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016764.214805:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332678656/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016764.214819:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734016764.214820:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016764.214824:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016764.214827:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734016764.214831:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016764/real 1734016764] req@ffff8800a73cfb80 x1818247332678144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016764.214833:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1026 00000100:00080000:3.0:1734016764.214837:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016764.214842:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734016764.214849:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332678144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734016764.214849:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734016764.214862:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1734016764.214862:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332678784/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734016764.214866:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016764.214872:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332678144:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016764.214881:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332678272:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016764.214905:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016764.214916:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016764/real 1734016764] req@ffff8800a73cd880 x1818247332678272/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016764.214933:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332678272/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016764.214947:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016764.214951:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016764.214956:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332678272:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016764.214966:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332678400:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016764.214989:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016764.214998:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016764/real 1734016764] req@ffff8800a73cf480 x1818247332678400/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016764.215009:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332678400/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016764.215017:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016764.215021:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016764.215024:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332678400:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016764.215073:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332678528:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016764.215091:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016764.215099:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016764/real 1734016764] req@ffff8800a73cdf80 x1818247332678528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016764.215110:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332678528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016764.215119:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016764.215121:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016764.215124:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332678528:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016764.215143:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332678656:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016764.215155:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016764.215161:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016764/real 1734016764] req@ffff8800a73ced80 x1818247332678656/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016764.215171:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332678656/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016764.215179:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016764.215181:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016764.215184:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332678656:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016764.215189:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332678784:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016764.215205:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016764.215211:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016764/real 1734016764] req@ffff8800a73cf800 x1818247332678784/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016764.215267:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332678784/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016764.215276:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016764.215279:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016764.215282:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332678784:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734016764.236511:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016765.238473:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016766.240477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016767.242374:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016768.244462:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016769.222397:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016769.222409:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016769.222415:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016769.222429:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1031 00000100:00080000:3.0:1734016769.222443:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016769.222453:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016769.222504:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332678912/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016769.222546:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016769.222553:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016769.222556:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016769.222563:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1031 00000100:00080000:3.0:1734016769.222568:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016769.222575:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016769.222590:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016769.222611:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332679040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016769.222623:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332678912:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016769.222627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016769.222632:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016769.222634:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016769.222640:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1031 00000100:00080000:3.0:1734016769.222644:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016769.222650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016769.222657:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016769.222669:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332679168/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016769.222681:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016769.222686:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016769.222688:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016769.222693:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1031 00000100:00080000:3.0:1734016769.222713:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734016769.222717:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016769.222719:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016769.222726:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016769.222740:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332679296/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016769.222749:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016769/real 1734016769] req@ffff8800a73cd500 x1818247332678912/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016769.222753:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016769.222759:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016769.222762:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016769.222767:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1031 00000100:00080000:3.0:1734016769.222772:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734016769.222775:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332678912/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016769.222777:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016769.222785:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1734016769.222790:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016769.222795:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734016769.222798:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332679424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016769.222802:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332678912:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016769.222812:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016769.222817:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016769.222819:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016769.222825:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1031 00000100:00080000:3.0:1734016769.222829:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016769.222834:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734016769.222835:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332679040:192.168.201.152@tcp:38:kworker.0 00000008:00100000:3.0:1734016769.222842:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016769.222855:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332679552/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734016769.222865:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016769.222870:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734016769.222876:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016769/real 1734016769] req@ffff8800a73cc700 x1818247332679040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016769.222876:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016769.222879:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016769.222885:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1031 00000100:00080000:3.0:1734016769.222889:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016769.222895:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016769.222902:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734016769.222908:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332679040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016769.222915:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332679680/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734016769.222922:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016769.222926:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016769.222931:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332679040:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016769.222939:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332679168:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016769.223011:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016769.223025:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016769/real 1734016769] req@ffff8800a73cea00 x1818247332679168/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016769.223044:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332679168/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016769.223057:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016769.223060:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016769.223064:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332679168:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016769.223071:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332679296:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016769.223092:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016769.223100:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016769/real 1734016769] req@ffff8800a73cdc00 x1818247332679296/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016769.223111:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332679296/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016769.223119:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016769.223122:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016769.223125:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332679296:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016769.223131:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332679424:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016769.223144:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016769.223150:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016769/real 1734016769] req@ffff8800a73cf100 x1818247332679424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016769.223161:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332679424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016769.223169:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016769.223172:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016769.223174:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332679424:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016769.223193:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332679552:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016769.223206:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016769.223212:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016769/real 1734016769] req@ffff8800a73cce00 x1818247332679552/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016769.223269:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332679552/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016769.223278:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016769.223280:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016769.223283:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332679552:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016769.223289:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332679680:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016769.223308:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016769.223315:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016769/real 1734016769] req@ffff8800a73cc380 x1818247332679680/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016769.223326:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332679680/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016769.223334:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016769.223336:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016769.223339:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332679680:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016769.246514:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016770.248490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016771.250511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016772.252524:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016773.254476:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016774.230368:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016774.230381:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016774.230386:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016774.230401:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1036 00000100:00080000:3.0:1734016774.230414:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016774.230425:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016774.230476:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332679808/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016774.230526:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016774.230532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016774.230536:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016774.230543:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1036 00000100:00080000:3.0:1734016774.230548:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016774.230555:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016774.230570:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016774.230593:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332679936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016774.230604:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332679808:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016774.230608:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016774.230613:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016774.230615:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016774.230621:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1036 00000100:00080000:3.0:1734016774.230624:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016774.230630:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016774.230636:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016774.230649:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332680064/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016774.230660:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016774.230665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016774.230668:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016774.230672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1036 00000100:00080000:3.0:1734016774.230676:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016774.230682:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016774.230688:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734016774.230698:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734016774.230700:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332680192/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016774.230712:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016774.230717:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016774.230719:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016774.230725:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1036 00000100:00100000:1.0:1734016774.230729:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016774/real 1734016774] req@ffff8800a73ce680 x1818247332679808/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016774.230730:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016774.230736:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016774.230743:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734016774.230756:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332679808/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016774.230756:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332680320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016774.230770:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1734016774.230773:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016774.230776:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1734016774.230777:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734016774.230778:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734016774.230784:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332679808:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016774.230784:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1036 00000100:00080000:3.0:1734016774.230789:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016774.230794:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016774.230800:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016774.230812:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332680448/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016774.230818:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332679936:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734016774.230826:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016774.230832:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016774.230835:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016774.230840:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1036 00000100:00080000:3.0:1734016774.230844:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734016774.230846:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016774.230850:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734016774.230856:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016774/real 1734016774] req@ffff8800a73cc380 x1818247332679936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734016774.230857:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016774.230908:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332680576/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016774.230909:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332679936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016774.230922:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016774.230927:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016774.230932:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332679936:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016774.230941:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332680064:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016774.230973:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016774.230986:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016774/real 1734016774] req@ffff8800a73cce00 x1818247332680064/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016774.231005:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332680064/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016774.231015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016774.231018:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016774.231022:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332680064:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016774.231029:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332680192:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016774.231049:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016774.231056:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016774/real 1734016774] req@ffff8800a73cf100 x1818247332680192/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016774.231067:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332680192/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016774.231075:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016774.231078:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016774.231081:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332680192:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016774.231087:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332680320:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016774.231101:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016774.231107:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016774/real 1734016774] req@ffff8800a73cdc00 x1818247332680320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016774.231117:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332680320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016774.231125:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016774.231128:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016774.231131:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332680320:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016774.231150:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332680448:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016774.231163:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016774.231169:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016774/real 1734016774] req@ffff8800a73cea00 x1818247332680448/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016774.231178:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332680448/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016774.231187:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016774.231189:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016774.231192:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332680448:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016774.231197:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332680576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016774.231213:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016774.231266:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016774/real 1734016774] req@ffff8800a73cc700 x1818247332680576/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016774.231278:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332680576/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016774.231286:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016774.231289:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016774.231292:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332680576:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734016774.256477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016775.258509:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016776.260464:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016777.262508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016778.264457:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016779.238378:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016779.238389:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016779.238395:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016779.238409:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1041 00000100:00080000:3.0:1734016779.238421:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016779.238431:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016779.238481:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332680704/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016779.238523:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016779.238530:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016779.238533:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016779.238541:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1041 00000100:00080000:3.0:1734016779.238545:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016779.238552:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016779.238567:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016779.238586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332680832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016779.238603:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734016779.238607:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332680704:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016779.238608:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016779.238610:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016779.238616:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1041 00000100:00080000:3.0:1734016779.238620:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016779.238625:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016779.238633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016779.238646:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332680960/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016779.238659:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016779.238663:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016779.238666:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016779.238671:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1041 00000100:00080000:3.0:1734016779.238674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016779.238679:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016779.238686:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016779.238698:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332681088/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734016779.238699:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016779.238711:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016779.238715:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016779.238718:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016779.238723:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1041 00000100:00080000:3.0:1734016779.238728:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734016779.238729:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016779/real 1734016779] req@ffff8800a73cd500 x1818247332680704/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016779.238733:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016779.238740:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016779.238754:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332681216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016779.238756:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332680704/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016779.238768:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1734016779.238771:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016779.238812:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016779.238819:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332680704:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016779.238828:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016779.238831:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016779.238837:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1041 00000100:00080000:3.0:1734016779.238841:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016779.238847:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734016779.238854:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332680832:192.168.201.152@tcp:38:kworker.0 00000008:00100000:3.0:1734016779.238854:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016779.238869:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332681344/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016779.238884:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734016779.238886:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016779.238889:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016779.238892:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016779.238898:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1041 00000100:00100000:1.0:1734016779.238899:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016779/real 1734016779] req@ffff8800a73cf800 x1818247332680832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016779.238902:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016779.238908:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016779.238916:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734016779.238929:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332680832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016779.238929:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332681472/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734016779.238943:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016779.238947:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016779.238953:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332680832:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016779.238961:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332680960:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016779.238988:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016779.239000:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016779/real 1734016779] req@ffff8800a73ced80 x1818247332680960/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016779.239018:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332680960/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016779.239031:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016779.239035:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016779.239040:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332680960:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016779.239047:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332681088:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016779.239068:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016779.239076:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016779/real 1734016779] req@ffff8800a73cdf80 x1818247332681088/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016779.239087:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332681088/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016779.239095:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016779.239099:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016779.239102:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332681088:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016779.239107:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332681216:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016779.239121:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016779.239126:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016779/real 1734016779] req@ffff8800a73cf480 x1818247332681216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016779.239136:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332681216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016779.239145:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016779.239147:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016779.239150:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332681216:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016779.239168:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332681344:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016779.239182:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016779.239188:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016779/real 1734016779] req@ffff8800a73cd880 x1818247332681344/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016779.239197:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332681344/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016779.239205:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016779.239208:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016779.239211:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332681344:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016779.239260:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332681472:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016779.239281:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016779.239289:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016779/real 1734016779] req@ffff8800a73cfb80 x1818247332681472/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016779.239300:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332681472/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016779.239309:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016779.239311:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016779.239314:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332681472:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016779.266487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016780.268478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016781.270491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016782.272473:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016783.274515:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016784.246390:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016784.246401:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016784.246407:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016784.246419:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1046 00000100:00080000:3.0:1734016784.246432:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016784.246441:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016784.246492:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332681600/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016784.246535:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016784.246541:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016784.246545:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016784.246552:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1046 00000100:00080000:3.0:1734016784.246557:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016784.246564:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016784.246578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016784.246599:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332681728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016784.246613:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332681600:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016784.246615:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016784.246619:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016784.246622:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016784.246627:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1046 00000100:00080000:3.0:1734016784.246632:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016784.246637:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016784.246644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016784.246657:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332681856/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016784.246670:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016784.246675:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016784.246677:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016784.246682:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1046 00000100:00080000:3.0:1734016784.246686:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016784.246726:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016784.246734:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016784.246750:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332681984/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734016784.246755:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016784.246764:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016784.246769:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016784.246772:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016784.246777:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1046 00000100:00080000:3.0:1734016784.246781:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016784.246787:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734016784.246788:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016784/real 1734016784] req@ffff8800a73cca80 x1818247332681600/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734016784.246794:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016784.246807:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332682112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016784.246815:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332681600/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016784.246822:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016784.246827:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1734016784.246830:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016784.246830:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734016784.246835:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734016784.246837:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1046 00000100:00100000:1.0:1734016784.246842:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332681600:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016784.246842:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016784.246847:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016784.246854:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016784.246864:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332682240/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016784.246875:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332681728:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734016784.246878:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016784.246883:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016784.246886:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016784.246891:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1046 00000100:00080000:3.0:1734016784.246895:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016784.246900:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734016784.246905:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734016784.246907:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734016784.246916:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016784/real 1734016784] req@ffff8800a73cfb80 x1818247332681728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016784.246920:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332682368/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016784.246936:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332681728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016784.246949:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016784.246952:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016784.246958:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332681728:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016784.246967:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332681856:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016784.246995:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016784.247008:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016784/real 1734016784] req@ffff8800a73cd880 x1818247332681856/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016784.247025:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332681856/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016784.247034:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016784.247037:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016784.247041:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332681856:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016784.247047:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332681984:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016784.247064:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016784.247071:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016784/real 1734016784] req@ffff8800a73cf480 x1818247332681984/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016784.247081:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332681984/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016784.247089:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016784.247093:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016784.247096:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332681984:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016784.247102:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332682112:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016784.247115:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016784.247121:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016784/real 1734016784] req@ffff8800a73cdf80 x1818247332682112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016784.247131:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332682112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016784.247139:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016784.247142:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016784.247145:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332682112:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016784.247163:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332682240:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016784.247176:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016784.247182:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016784/real 1734016784] req@ffff8800a73ced80 x1818247332682240/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016784.247192:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332682240/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016784.247200:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016784.247202:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016784.247205:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332682240:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016784.247210:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332682368:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016784.247274:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016784.247282:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016784/real 1734016784] req@ffff8800a73cf800 x1818247332682368/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016784.247294:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332682368/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016784.247303:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016784.247305:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016784.247308:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332682368:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734016784.276475:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016785.278508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016786.280497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016787.282494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016788.284514:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016789.254364:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016789.254375:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016789.254380:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016789.254394:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1051 00000100:00080000:3.0:1734016789.254406:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016789.254416:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016789.254464:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332682496/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016789.254512:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016789.254519:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016789.254522:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016789.254529:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1051 00000100:00080000:3.0:1734016789.254534:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016789.254541:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016789.254556:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016789.254577:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332682624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016789.254590:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332682496:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016789.254592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016789.254597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016789.254599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016789.254641:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1051 00000100:00080000:3.0:1734016789.254646:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016789.254652:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016789.254659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016789.254675:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332682752/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016789.254688:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016789.254692:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016789.254695:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016789.254699:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1051 00000100:00080000:3.0:1734016789.254703:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016789.254708:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016789.254715:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016789.254727:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332682880/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734016789.254732:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016789.254740:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016789.254745:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016789.254747:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016789.254753:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1051 00000100:00080000:3.0:1734016789.254756:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734016789.254762:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016789/real 1734016789] req@ffff8800a73cd500 x1818247332682496/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016789.254762:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016789.254769:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016789.254782:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332683008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016789.254788:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332682496/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016789.254796:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016789.254802:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1734016789.254803:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016789.254804:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734016789.254808:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734016789.254810:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1051 00000100:00080000:3.0:1734016789.254814:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734016789.254815:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332682496:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016789.254819:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016789.254826:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016789.254838:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332683136/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016789.254848:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332682624:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734016789.254852:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016789.254857:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016789.254860:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016789.254866:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1051 00000100:00080000:3.0:1734016789.254870:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016789.254876:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734016789.254877:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734016789.254883:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734016789.254888:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016789/real 1734016789] req@ffff8800a73cc700 x1818247332682624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016789.254898:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332683264/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016789.254918:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332682624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016789.254931:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016789.254935:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016789.254940:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332682624:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016789.254949:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332682752:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016789.254978:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016789.254991:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016789/real 1734016789] req@ffff8800a73cea00 x1818247332682752/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016789.255007:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332682752/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016789.255016:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016789.255018:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016789.255022:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332682752:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016789.255028:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332682880:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016789.255045:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016789.255052:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016789/real 1734016789] req@ffff8800a73cdc00 x1818247332682880/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016789.255062:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332682880/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016789.255070:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016789.255073:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016789.255076:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332682880:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016789.255082:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332683008:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016789.255095:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016789.255101:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016789/real 1734016789] req@ffff8800a73cf100 x1818247332683008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016789.255111:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332683008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016789.255119:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016789.255122:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016789.255125:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332683008:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016789.255143:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332683136:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016789.255155:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016789.255161:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016789/real 1734016789] req@ffff8800a73cce00 x1818247332683136/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016789.255171:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332683136/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016789.255179:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016789.255182:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016789.255185:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332683136:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016789.255190:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332683264:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016789.255205:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016789.255211:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016789/real 1734016789] req@ffff8800a73cc380 x1818247332683264/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016789.255268:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332683264/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016789.255276:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016789.255278:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016789.255282:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332683264:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016789.286530:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016790.288528:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016791.290488:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016792.292516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016793.294463:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016794.262352:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016794.262363:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016794.262368:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016794.262383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1056 00000100:00080000:3.0:1734016794.262394:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016794.262404:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016794.262452:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332683392/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016794.262494:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016794.262500:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016794.262504:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016794.262511:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1056 00000100:00080000:3.0:1734016794.262516:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016794.262557:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016794.262572:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016794.262595:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332683520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016794.262603:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332683392:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016794.262614:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016794.262618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016794.262621:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016794.262627:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1056 00000100:00080000:3.0:1734016794.262631:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016794.262636:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016794.262643:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016794.262656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332683648/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016794.262668:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016794.262672:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016794.262675:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016794.262679:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1056 00000100:00080000:3.0:1734016794.262683:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016794.262688:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734016794.262695:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734016794.262695:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016794.262707:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332683776/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016794.262719:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734016794.262724:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016794/real 1734016794] req@ffff8800a73ce680 x1818247332683392/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016794.262725:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016794.262728:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016794.262734:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1056 00000100:00080000:3.0:1734016794.262738:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016794.262743:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734016794.262750:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332683392/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734016794.262750:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016794.262763:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332683904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734016794.262765:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016794.262770:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016794.262777:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332683392:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016794.262777:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016794.262782:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016794.262785:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016794.262790:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1056 00000100:00080000:3.0:1734016794.262794:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016794.262799:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016794.262806:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734016794.262810:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332683520:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734016794.262818:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332684032/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016794.262831:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016794.262837:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1734016794.262838:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016794.262840:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016794.262845:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1056 00000100:00100000:1.0:1734016794.262849:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016794/real 1734016794] req@ffff8800a73cc380 x1818247332683520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016794.262850:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016794.262856:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016794.262863:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734016794.262869:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332683520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016794.262876:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332684160/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734016794.262882:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016794.262886:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016794.262891:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332683520:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016794.262900:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332683648:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016794.262927:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016794.262939:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016794/real 1734016794] req@ffff8800a73cce00 x1818247332683648/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016794.262958:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332683648/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016794.262971:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016794.262975:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016794.262980:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332683648:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016794.262989:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332683776:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016794.263016:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016794.263025:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016794/real 1734016794] req@ffff8800a73cf100 x1818247332683776/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016794.263037:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332683776/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016794.263045:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016794.263048:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016794.263051:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332683776:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016794.263057:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332683904:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016794.263072:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016794.263078:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016794/real 1734016794] req@ffff8800a73cdc00 x1818247332683904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016794.263088:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332683904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016794.263096:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016794.263099:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016794.263102:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332683904:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016794.263120:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332684032:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016794.263133:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016794.263139:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016794/real 1734016794] req@ffff8800a73cea00 x1818247332684032/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016794.263149:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332684032/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016794.263158:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016794.263160:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016794.263163:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332684032:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016794.263168:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332684160:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016794.263184:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016794.263190:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016794/real 1734016794] req@ffff8800a73cc700 x1818247332684160/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016794.263201:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332684160/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016794.263208:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016794.263210:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016794.263213:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332684160:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734016794.296499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016795.298467:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016796.300526:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016797.302522:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016798.304534:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016799.270374:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016799.270385:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016799.270391:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016799.270404:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1061 00000100:00080000:3.0:1734016799.270416:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016799.270427:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016799.270505:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332684288/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016799.270544:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016799.270551:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016799.270554:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016799.270561:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1061 00000100:00080000:3.0:1734016799.270566:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016799.270574:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016799.270589:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016799.270610:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332684416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016799.270626:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734016799.270629:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332684288:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016799.270631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016799.270634:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016799.270640:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1061 00000100:00080000:3.0:1734016799.270644:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016799.270650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016799.270657:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016799.270670:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332684544/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016799.270683:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016799.270688:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016799.270690:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016799.270695:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1061 00000100:00080000:3.0:1734016799.270699:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016799.270704:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016799.270710:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016799.270722:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332684672/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734016799.270723:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016799.270735:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016799.270740:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016799.270742:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016799.270747:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1061 00000100:00100000:1.0:1734016799.270751:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016799/real 1734016799] req@ffff8800a73cd500 x1818247332684288/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016799.270752:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016799.270758:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016799.270765:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734016799.270778:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332684288/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016799.270778:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332684800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016799.270792:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1734016799.270794:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016799.270796:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1734016799.270799:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734016799.270800:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734016799.270806:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332684288:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016799.270806:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1061 00000100:00080000:3.0:1734016799.270811:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016799.270815:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016799.270822:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016799.270833:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332684928/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016799.270838:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332684416:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734016799.270847:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016799.270853:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016799.270856:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016799.270861:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1061 00000800:00000100:1.0:1734016799.270866:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016799.270866:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016799.270871:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734016799.270877:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016799/real 1734016799] req@ffff8800a73cf800 x1818247332684416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734016799.270878:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016799.270892:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332685056/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016799.270907:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332684416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016799.270921:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016799.270924:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016799.270930:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332684416:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016799.270938:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332684544:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016799.270969:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016799.270982:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016799/real 1734016799] req@ffff8800a73ced80 x1818247332684544/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016799.270996:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332684544/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016799.271005:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016799.271007:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016799.271011:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332684544:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016799.271017:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332684672:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016799.271035:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016799.271041:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016799/real 1734016799] req@ffff8800a73cdf80 x1818247332684672/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016799.271052:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332684672/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016799.271060:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016799.271063:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016799.271066:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332684672:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016799.271072:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332684800:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016799.271085:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016799.271091:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016799/real 1734016799] req@ffff8800a73cf480 x1818247332684800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016799.271101:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332684800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016799.271110:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016799.271112:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016799.271115:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332684800:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016799.271132:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332684928:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016799.271146:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016799.271152:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016799/real 1734016799] req@ffff8800a73cd880 x1818247332684928/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016799.271161:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332684928/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016799.271169:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016799.271172:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016799.271175:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332684928:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016799.271180:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332685056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016799.271195:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016799.271202:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016799/real 1734016799] req@ffff8800a73cfb80 x1818247332685056/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016799.271212:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332685056/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016799.271263:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016799.271266:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016799.271269:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332685056:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016799.306513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016800.308473:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016801.310501:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016802.312423:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016803.314526:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016804.278402:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016804.278412:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016804.278417:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016804.278432:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1066 00000100:00080000:3.0:1734016804.278444:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016804.278454:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016804.278502:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332685184/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016804.278551:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016804.278558:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016804.278561:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016804.278569:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1066 00000100:00080000:3.0:1734016804.278574:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016804.278581:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016804.278596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016804.278619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332685312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016804.278631:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332685184:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016804.278635:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016804.278640:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016804.278642:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016804.278648:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1066 00000100:00080000:3.0:1734016804.278652:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016804.278657:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016804.278664:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016804.278677:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332685440/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016804.278689:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016804.278693:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016804.278696:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016804.278700:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1066 00000100:00080000:3.0:1734016804.278717:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734016804.278724:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016804.278724:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016804.278732:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016804.278744:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332685568/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016804.278756:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016804/real 1734016804] req@ffff8800a73cca80 x1818247332685184/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016804.278757:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016804.278763:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016804.278766:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016804.278772:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1066 00000100:00080000:3.0:1734016804.278775:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016804.278781:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734016804.278782:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332685184/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734016804.278788:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1734016804.278797:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016804.278801:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734016804.278802:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332685696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016804.278808:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332685184:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016804.278816:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016804.278821:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016804.278823:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016804.278829:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1066 00000100:00080000:3.0:1734016804.278833:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016804.278838:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734016804.278840:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332685312:192.168.201.152@tcp:38:kworker.0 00000008:00100000:3.0:1734016804.278845:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016804.278857:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332685824/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734016804.278867:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016804.278873:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734016804.278879:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016804/real 1734016804] req@ffff8800a73cfb80 x1818247332685312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016804.278879:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016804.278882:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016804.278887:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1066 00000100:00080000:3.0:1734016804.278891:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016804.278897:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734016804.278898:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332685312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734016804.278904:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734016804.278912:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016804.278916:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734016804.278918:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332685952/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016804.278922:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332685312:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016804.278931:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332685440:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016804.278957:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016804.278968:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016804/real 1734016804] req@ffff8800a73cd880 x1818247332685440/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016804.278987:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332685440/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016804.279000:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016804.279003:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016804.279008:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332685440:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016804.279017:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332685568:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016804.279038:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016804.279046:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016804/real 1734016804] req@ffff8800a73cf480 x1818247332685568/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016804.279057:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332685568/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016804.279065:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016804.279069:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016804.279072:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332685568:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016804.279078:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332685696:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016804.279091:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016804.279097:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016804/real 1734016804] req@ffff8800a73cdf80 x1818247332685696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016804.279107:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332685696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016804.279115:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016804.279118:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016804.279121:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332685696:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016804.279139:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332685824:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016804.279152:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016804.279158:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016804/real 1734016804] req@ffff8800a73ced80 x1818247332685824/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016804.279167:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332685824/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016804.279175:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016804.279178:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016804.279181:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332685824:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016804.279186:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332685952:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016804.279202:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016804.279208:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016804/real 1734016804] req@ffff8800a73cf800 x1818247332685952/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016804.279266:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332685952/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016804.279274:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016804.279277:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016804.279280:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332685952:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734016804.316538:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016805.318599:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016806.320515:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016807.322515:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016808.324550:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016809.286417:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016809.286430:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016809.286436:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016809.286450:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1071 00000100:00080000:3.0:1734016809.286463:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016809.286473:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016809.286524:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332686080/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016809.286567:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016809.286574:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016809.286577:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016809.286584:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1071 00000100:00080000:3.0:1734016809.286589:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016809.286596:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016809.286613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016809.286632:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332686208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016809.286648:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332686080:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016809.286649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016809.286654:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016809.286656:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016809.286661:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1071 00000100:00080000:3.0:1734016809.286665:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016809.286671:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016809.286677:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016809.286690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332686336/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016809.286703:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016809.286707:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016809.286710:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016809.286715:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1071 00000100:00080000:3.0:1734016809.286718:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016809.286723:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016809.286730:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734016809.286743:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734016809.286743:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332686464/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016809.286755:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016809.286760:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016809.286762:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016809.286768:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1071 00000100:00080000:3.0:1734016809.286773:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734016809.286775:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016809/real 1734016809] req@ffff8800a73cd500 x1818247332686080/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016809.286779:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016809.286786:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016809.286800:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332686592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016809.286801:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332686080/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016809.286814:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1734016809.286817:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734016809.286819:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016809.286821:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734016809.286822:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016809.286828:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332686080:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016809.286828:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1071 00000100:00080000:3.0:1734016809.286832:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016809.286838:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016809.286844:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016809.286855:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332686720/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016809.286861:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332686208:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734016809.286870:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016809.286875:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016809.286878:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016809.286883:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1071 00000100:00080000:3.0:1734016809.286888:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734016809.286889:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016809.286893:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734016809.286900:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016809/real 1734016809] req@ffff8800a73cc700 x1818247332686208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734016809.286901:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016809.286915:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332686848/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016809.286931:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332686208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016809.286943:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016809.286946:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016809.286951:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332686208:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016809.286960:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332686336:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016809.286987:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016809.287000:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016809/real 1734016809] req@ffff8800a73cea00 x1818247332686336/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016809.287017:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332686336/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016809.287029:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016809.287032:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016809.287036:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332686336:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016809.287043:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332686464:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016809.287062:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016809.287069:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016809/real 1734016809] req@ffff8800a73cdc00 x1818247332686464/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016809.287079:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332686464/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016809.287087:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016809.287090:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016809.287093:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332686464:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016809.287099:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332686592:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016809.287112:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016809.287118:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016809/real 1734016809] req@ffff8800a73cf100 x1818247332686592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016809.287128:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332686592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016809.287137:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016809.287139:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016809.287142:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332686592:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016809.287160:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332686720:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016809.287173:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016809.287179:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016809/real 1734016809] req@ffff8800a73cce00 x1818247332686720/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016809.287189:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332686720/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016809.287197:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016809.287199:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016809.287203:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332686720:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016809.287207:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332686848:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016809.287307:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016809.287317:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016809/real 1734016809] req@ffff8800a73cc380 x1818247332686848/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016809.287329:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332686848/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016809.287338:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016809.287340:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016809.287343:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332686848:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016809.326500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734016810.328451:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016811.330479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016812.332517:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016813.334479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016814.294343:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016814.294354:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016814.294358:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016814.294369:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1076 00000100:00080000:3.0:1734016814.294380:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016814.294388:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016814.294425:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332686976/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016814.294462:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016814.294466:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016814.294468:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016814.294473:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1076 00000100:00080000:3.0:1734016814.294476:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016814.294480:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016814.294490:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016814.294502:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332687104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016814.294512:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016814.294515:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016814.294517:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016814.294521:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1076 00000100:00080000:3.0:1734016814.294523:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016814.294528:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016814.294533:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016814.294541:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332687232/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016814.294550:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016814.294553:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016814.294555:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016814.294558:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1076 00000100:00080000:3.0:1734016814.294561:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016814.294564:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016814.294568:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016814.294576:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332687360/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016814.294584:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016814.294587:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016814.294588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016814.294592:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1076 00000100:00080000:3.0:1734016814.294594:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016814.294597:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016814.294602:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016814.294609:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332687488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016814.294619:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016814.294623:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016814.294624:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016814.294628:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1076 00000100:00100000:1.0:1734016814.294631:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332686976:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016814.294631:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016814.294634:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016814.294639:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016814.294646:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332687616/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016814.294654:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016814.294657:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016814.294659:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016814.294662:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1076 00000100:00080000:3.0:1734016814.294665:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016814.294668:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016814.294672:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016814.294680:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332687744/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734016814.294732:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016814.294763:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016814/real 1734016814] req@ffff8800a73ce680 x1818247332686976/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016814.294786:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332686976/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016814.294798:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016814.294802:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016814.294807:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332686976:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734016814.294815:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332687104:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016814.294835:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016814.294842:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016814/real 1734016814] req@ffff8800a73cc380 x1818247332687104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016814.294855:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332687104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016814.294863:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016814.294866:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016814.294870:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332687104:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016814.294876:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332687232:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016814.294891:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016814.294897:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016814/real 1734016814] req@ffff8800a73cce00 x1818247332687232/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016814.294907:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332687232/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016814.294915:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016814.294917:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016814.294920:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332687232:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016814.294926:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332687360:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016814.294939:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016814.294946:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016814/real 1734016814] req@ffff8800a73cf100 x1818247332687360/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016814.294956:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332687360/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016814.294963:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016814.294967:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016814.294970:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332687360:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016814.295006:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332687488:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016814.295021:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016814.295027:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016814/real 1734016814] req@ffff8800a73cdc00 x1818247332687488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016814.295038:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332687488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016814.295046:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016814.295049:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016814.295052:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332687488:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016814.295057:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332687616:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016814.295070:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016814.295076:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016814/real 1734016814] req@ffff8800a73cea00 x1818247332687616/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016814.295086:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332687616/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016814.295093:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016814.295095:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016814.295098:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332687616:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016814.295104:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332687744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016814.295119:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016814.295125:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016814/real 1734016814] req@ffff8800a73cc700 x1818247332687744/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016814.295136:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332687744/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016814.295143:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016814.295146:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016814.295149:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332687744:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016814.336494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016815.338483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016816.340483:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016817.342509:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016818.344541:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016819.302397:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016819.302407:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016819.302412:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016819.302423:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1081 00000100:00080000:3.0:1734016819.302434:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016819.302442:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016819.302480:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332687872/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016819.302518:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016819.302522:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016819.302524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016819.302529:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1081 00000100:00080000:3.0:1734016819.302532:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016819.302536:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016819.302546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016819.302556:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332688000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016819.302567:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016819.302570:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016819.302572:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016819.302576:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1081 00000100:00080000:3.0:1734016819.302578:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016819.302582:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016819.302587:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016819.302594:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332688128/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016819.302603:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016819.302606:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016819.302608:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016819.302612:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1081 00000100:00080000:3.0:1734016819.302614:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016819.302618:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016819.302622:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016819.302630:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332688256/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016819.302640:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016819.302643:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016819.302645:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016819.302648:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1081 00000100:00080000:3.0:1734016819.302651:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016819.302655:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016819.302660:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734016819.302665:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332687872:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734016819.302668:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332688384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016819.302677:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016819.302680:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016819.302681:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016819.302686:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1081 00000100:00080000:3.0:1734016819.302688:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016819.302691:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016819.302696:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016819.302703:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332688512/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016819.302711:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016819.302715:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016819.302716:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016819.302720:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1081 00000100:00080000:3.0:1734016819.302722:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016819.302726:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016819.302730:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016819.302739:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332688640/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016819.302791:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332688000:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016819.302818:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332688128:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016819.302847:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332688256:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016819.302863:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332688384:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016819.302881:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332688512:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016819.302897:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332688640:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016819.346448:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016820.348505:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016821.350495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734016822.308662:0:1958:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:3.0:1734016822.308718:0:1958:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734016822.308738:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734016822.308777:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734016822.308786:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734016822.308793:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734016822.308800:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734016822.308806:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734016822.308814:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734016822.308937:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016819/real 1734016822] req@ffff8800a73cdf80 x1818247332688256/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016822.308985:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332688256/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016822.308999:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016822.309004:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016822.309010:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332688256:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016822.309036:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016819/real 1734016822] req@ffff8800a73cf480 x1818247332688384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016822.309089:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332688384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016822.309100:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016822.309105:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016822.309111:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332688384:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016822.309124:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016819/real 1734016822] req@ffff8800a73cd880 x1818247332688512/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016822.309142:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332688512/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016822.309154:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016822.309159:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016822.309164:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332688512:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016822.309175:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016819/real 1734016822] req@ffff8800a73cfb80 x1818247332688640/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016822.309206:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332688640/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016822.309358:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016822.309364:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016822.309371:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332688640:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016822.309383:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016819/real 1734016822] req@ffff8800a73cd500 x1818247332687872/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016822.309406:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332687872/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016822.309418:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016822.309422:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016822.309428:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332687872:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734016822.309439:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016819/real 1734016822] req@ffff8800a73cf800 x1818247332688000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016822.309458:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332688000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016822.309471:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016822.309476:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016822.309482:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332688000:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016822.309493:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016819/real 1734016822] req@ffff8800a73ced80 x1818247332688128/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016822.309523:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332688128/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016822.309545:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016822.309550:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016822.309556:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332688128:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016822.352393:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016823.354523:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016824.310330:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016824.310340:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016824.310345:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016824.310356:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1086 00000100:00080000:3.0:1734016824.310367:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016824.310375:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016824.310410:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332688768/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016824.310446:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016824.310451:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016824.310453:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016824.310457:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1086 00000100:00080000:3.0:1734016824.310460:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016824.310465:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016824.310474:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016824.310487:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332688896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016824.310497:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016824.310500:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016824.310502:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016824.310506:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1086 00000100:00080000:3.0:1734016824.310508:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016824.310511:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016824.310516:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016824.310525:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332689024/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016824.310533:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016824.310539:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016824.310541:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016824.310544:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1086 00000100:00080000:3.0:1734016824.310547:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016824.310550:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016824.310554:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016824.310562:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332689152/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016824.310570:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016824.310573:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016824.310574:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016824.310578:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1086 00000100:00080000:3.0:1734016824.310580:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016824.310583:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016824.310587:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016824.310595:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332689280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016824.310605:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016824.310607:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1734016824.310608:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332688768:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016824.310609:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016824.310613:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1086 00000100:00080000:3.0:1734016824.310616:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016824.310619:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016824.310624:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016824.310632:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332689408/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016824.310641:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016824.310644:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016824.310645:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016824.310649:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1086 00000100:00080000:3.0:1734016824.310651:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016824.310655:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016824.310659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016824.310667:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332689536/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734016824.310708:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016824.310738:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016824/real 1734016824] req@ffff8800a73cca80 x1818247332688768/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016824.310760:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332688768/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016824.310774:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016824.310778:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016824.310783:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332688768:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734016824.310791:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332688896:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016824.310810:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016824.310817:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016824/real 1734016824] req@ffff8800a73ced80 x1818247332688896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016824.310829:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332688896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016824.310845:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016824.310848:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016824.310851:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332688896:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016824.310858:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332689024:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016824.310874:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016824.310880:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016824/real 1734016824] req@ffff8800a73cf800 x1818247332689024/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016824.310890:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332689024/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016824.310897:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016824.310900:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016824.310904:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332689024:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016824.310910:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332689152:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016824.310923:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016824.310929:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016824/real 1734016824] req@ffff8800a73cd500 x1818247332689152/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016824.310939:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332689152/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016824.310946:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016824.310948:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016824.310951:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332689152:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016824.310985:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332689280:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016824.310999:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016824.311054:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016824/real 1734016824] req@ffff8800a73cfb80 x1818247332689280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016824.311069:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332689280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016824.311078:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016824.311080:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016824.311084:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332689280:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016824.311090:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332689408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016824.311107:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016824.311114:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016824/real 1734016824] req@ffff8800a73cd880 x1818247332689408/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016824.311124:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332689408/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016824.311132:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016824.311135:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016824.311138:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332689408:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016824.311143:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332689536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016824.311156:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016824.311161:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016824/real 1734016824] req@ffff8800a73cf480 x1818247332689536/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016824.311173:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332689536/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016824.311181:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016824.311183:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016824.311186:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332689536:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016824.356451:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016825.358480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016826.360499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016827.362472:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016828.364458:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016829.318361:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016829.318370:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016829.318375:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016829.318387:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1091 00000100:00080000:3.0:1734016829.318397:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016829.318405:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016829.318444:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332689664/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016829.318480:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016829.318487:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016829.318489:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016829.318494:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1091 00000100:00080000:3.0:1734016829.318497:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016829.318501:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016829.318511:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016829.318522:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332689792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016829.318532:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016829.318535:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016829.318537:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016829.318541:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1091 00000100:00080000:3.0:1734016829.318544:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016829.318547:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016829.318552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016829.318560:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332689920/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016829.318568:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016829.318572:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016829.318573:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016829.318577:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1091 00000100:00080000:3.0:1734016829.318579:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016829.318582:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016829.318587:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016829.318595:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332690048/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016829.318605:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016829.318608:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016829.318610:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016829.318613:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1091 00000100:00080000:3.0:1734016829.318616:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016829.318619:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016829.318624:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734016829.318628:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332689664:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734016829.318632:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332690176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016829.318641:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016829.318644:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016829.318645:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016829.318649:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1091 00000100:00080000:3.0:1734016829.318652:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016829.318655:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016829.318659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016829.318667:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332690304/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016829.318675:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016829.318678:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016829.318680:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016829.318684:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1091 00000100:00080000:3.0:1734016829.318687:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016829.318690:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016829.318695:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016829.318704:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332690432/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734016829.318725:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016829.318754:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016829/real 1734016829] req@ffff8800a73cdf80 x1818247332689664/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016829.318777:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332689664/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016829.318790:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016829.318794:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016829.318800:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332689664:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734016829.318808:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332689792:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016829.318828:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016829.318835:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016829/real 1734016829] req@ffff8800a73cc700 x1818247332689792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016829.318846:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332689792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016829.318856:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016829.318859:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016829.318862:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332689792:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016829.318869:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332689920:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016829.318884:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016829.318890:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016829/real 1734016829] req@ffff8800a73cea00 x1818247332689920/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016829.318900:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332689920/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016829.318908:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016829.318911:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016829.318914:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332689920:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016829.318990:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332690048:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016829.319009:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016829.319017:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016829/real 1734016829] req@ffff8800a73cdc00 x1818247332690048/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016829.319028:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332690048/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016829.319037:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016829.319039:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016829.319042:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332690048:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016829.319050:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332690176:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016829.319063:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016829.319069:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016829/real 1734016829] req@ffff8800a73cf100 x1818247332690176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016829.319081:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332690176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016829.319089:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016829.319091:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016829.319094:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332690176:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016829.319100:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332690304:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016829.319112:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016829.319118:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016829/real 1734016829] req@ffff8800a73cce00 x1818247332690304/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016829.319128:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332690304/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016829.319135:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016829.319137:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016829.319140:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332690304:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016829.319146:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332690432:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016829.319160:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016829.319166:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016829/real 1734016829] req@ffff8800a73cc380 x1818247332690432/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016829.319177:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332690432/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016829.319186:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016829.319188:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016829.319191:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332690432:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016829.366482:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016830.368483:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016831.370499:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016832.372479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016833.374525:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016834.326350:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016834.326360:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016834.326365:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016834.326376:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1096 00000100:00080000:3.0:1734016834.326386:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016834.326394:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016834.326429:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332690560/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016834.326467:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016834.326471:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016834.326473:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016834.326478:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1096 00000100:00080000:3.0:1734016834.326481:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016834.326485:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016834.326495:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016834.326508:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332690688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016834.326519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016834.326522:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016834.326524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016834.326528:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1096 00000100:00080000:3.0:1734016834.326530:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016834.326534:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016834.326538:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016834.326547:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332690816/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016834.326555:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016834.326559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016834.326560:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016834.326564:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1096 00000100:00080000:3.0:1734016834.326566:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016834.326569:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016834.326574:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016834.326581:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332690944/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016834.326589:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016834.326593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016834.326595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016834.326599:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1096 00000100:00080000:3.0:1734016834.326602:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016834.326605:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016834.326610:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016834.326618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332691072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016834.326624:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332690560:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016834.326628:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016834.326631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016834.326633:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016834.326637:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1096 00000100:00080000:3.0:1734016834.326639:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016834.326642:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016834.326647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016834.326654:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332691200/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016834.326662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016834.326665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016834.326667:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016834.326670:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1096 00000100:00080000:3.0:1734016834.326673:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016834.326676:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016834.326680:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016834.326688:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332691328/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734016834.326723:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016834.326753:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016834/real 1734016834] req@ffff8800a73ce680 x1818247332690560/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016834.326776:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332690560/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016834.326789:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016834.326793:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016834.326798:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332690560:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734016834.326806:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332690688:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016834.326825:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016834.326832:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016834/real 1734016834] req@ffff8800a73cc380 x1818247332690688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016834.326895:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332690688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016834.326905:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016834.326908:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016834.326912:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332690688:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016834.326918:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332690816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016834.326938:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016834.326945:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016834/real 1734016834] req@ffff8800a73cce00 x1818247332690816/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016834.326955:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332690816/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016834.326964:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016834.326967:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016834.326970:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332690816:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016834.326975:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332690944:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016834.326988:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016834.326994:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016834/real 1734016834] req@ffff8800a73cf100 x1818247332690944/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016834.327004:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332690944/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016834.327012:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016834.327014:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016834.327017:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332690944:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016834.327052:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332691072:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016834.327066:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016834.327073:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016834/real 1734016834] req@ffff8800a73cdc00 x1818247332691072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016834.327084:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332691072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016834.327092:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016834.327094:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016834.327097:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332691072:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016834.327103:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332691200:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016834.327116:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016834.327122:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016834/real 1734016834] req@ffff8800a73cea00 x1818247332691200/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016834.327132:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332691200/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016834.327139:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016834.327142:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016834.327145:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332691200:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016834.327151:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332691328:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016834.327164:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016834.327169:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016834/real 1734016834] req@ffff8800a73cc700 x1818247332691328/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016834.327181:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332691328/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016834.327189:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016834.327191:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016834.327194:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332691328:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016834.376441:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016835.378515:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016836.380480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016837.382535:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016838.384516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016839.334332:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016839.334342:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016839.334347:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016839.334358:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1101 00000100:00080000:3.0:1734016839.334369:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016839.334377:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016839.334414:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332691456/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016839.334451:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016839.334455:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016839.334457:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016839.334461:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1101 00000100:00080000:3.0:1734016839.334465:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016839.334469:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016839.334479:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016839.334489:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332691584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016839.334499:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016839.334503:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016839.334505:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016839.334508:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1101 00000100:00080000:3.0:1734016839.334512:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016839.334515:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016839.334520:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016839.334529:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332691712/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016839.334537:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016839.334540:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016839.334541:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016839.334545:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1101 00000100:00080000:3.0:1734016839.334548:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016839.334551:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016839.334555:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016839.334563:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332691840/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016839.334571:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016839.334574:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016839.334575:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016839.334579:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1101 00000100:00080000:3.0:1734016839.334581:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016839.334584:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016839.334589:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016839.334597:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332691968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016839.334602:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332691456:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016839.334613:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016839.334616:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016839.334617:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016839.334622:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1101 00000100:00080000:3.0:1734016839.334624:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016839.334627:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016839.334632:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016839.334639:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332692096/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016839.334647:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016839.334650:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016839.334652:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016839.334655:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1101 00000100:00080000:3.0:1734016839.334658:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016839.334661:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016839.334665:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016839.334674:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332692224/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734016839.334698:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016839.334728:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016839/real 1734016839] req@ffff8800a73cdf80 x1818247332691456/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016839.334796:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332691456/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016839.334810:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016839.334814:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016839.334819:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332691456:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734016839.334828:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332691584:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016839.334852:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016839.334860:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016839/real 1734016839] req@ffff8800a73cf480 x1818247332691584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016839.334872:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332691584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016839.334882:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016839.334885:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016839.334888:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332691584:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016839.334895:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332691712:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016839.334910:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016839.334916:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016839/real 1734016839] req@ffff8800a73cd880 x1818247332691712/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016839.334926:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332691712/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016839.334934:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016839.334937:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016839.334940:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332691712:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016839.334946:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332691840:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016839.334959:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016839.334965:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016839/real 1734016839] req@ffff8800a73cfb80 x1818247332691840/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016839.334974:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332691840/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016839.334982:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016839.334985:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016839.334989:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332691840:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016839.335022:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332691968:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016839.335036:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016839.335042:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016839/real 1734016839] req@ffff8800a73cd500 x1818247332691968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016839.335053:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332691968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016839.335062:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016839.335064:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016839.335067:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332691968:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016839.335073:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332692096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016839.335086:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016839.335092:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016839/real 1734016839] req@ffff8800a73cf800 x1818247332692096/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016839.335103:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332692096/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016839.335110:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016839.335112:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016839.335115:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332692096:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016839.335121:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332692224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016839.335133:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016839.335139:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016839/real 1734016839] req@ffff8800a73ced80 x1818247332692224/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016839.335150:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332692224/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016839.335159:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016839.335161:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016839.335164:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332692224:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016839.386503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016840.388512:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016841.390474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016842.392492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016843.394525:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016844.342353:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016844.342362:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016844.342367:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016844.342379:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1106 00000100:00080000:3.0:1734016844.342389:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016844.342397:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016844.342433:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332692352/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016844.342469:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016844.342474:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016844.342476:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016844.342480:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1106 00000100:00080000:3.0:1734016844.342483:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016844.342487:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016844.342497:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016844.342510:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332692480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016844.342520:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016844.342523:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016844.342525:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016844.342529:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1106 00000100:00080000:3.0:1734016844.342531:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016844.342535:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016844.342539:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016844.342548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332692608/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016844.342556:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016844.342560:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016844.342562:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016844.342565:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1106 00000100:00080000:3.0:1734016844.342568:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016844.342571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016844.342576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016844.342583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332692736/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016844.342591:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016844.342594:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016844.342596:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016844.342599:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1106 00000100:00080000:3.0:1734016844.342602:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016844.342605:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016844.342609:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016844.342618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332692864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016844.342625:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332692352:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016844.342628:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016844.342631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016844.342632:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016844.342636:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1106 00000100:00080000:3.0:1734016844.342639:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016844.342642:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016844.342647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016844.342655:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332692992/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016844.342663:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016844.342698:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016844.342700:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016844.342704:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1106 00000100:00080000:3.0:1734016844.342707:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016844.342710:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016844.342715:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016844.342725:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332693120/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734016844.342764:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016844.342795:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016844/real 1734016844] req@ffff8800a73cca80 x1818247332692352/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016844.342817:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332692352/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016844.342831:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016844.342834:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016844.342839:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332692352:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734016844.342848:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332692480:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016844.342867:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016844.342874:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016844/real 1734016844] req@ffff8800a73ced80 x1818247332692480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016844.342886:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332692480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016844.342896:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016844.342899:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016844.342902:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332692480:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016844.342909:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332692608:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016844.342924:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016844.342930:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016844/real 1734016844] req@ffff8800a73cf800 x1818247332692608/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016844.342940:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332692608/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016844.342948:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016844.342951:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016844.342954:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332692608:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016844.342960:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332692736:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016844.342974:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016844.342980:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016844/real 1734016844] req@ffff8800a73cd500 x1818247332692736/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016844.342990:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332692736/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016844.342997:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016844.342999:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016844.343002:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332692736:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016844.343036:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332692864:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016844.343051:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016844.343057:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016844/real 1734016844] req@ffff8800a73cfb80 x1818247332692864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016844.343068:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332692864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016844.343076:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016844.343078:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016844.343081:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332692864:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016844.343087:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332692992:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016844.343100:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016844.343105:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016844/real 1734016844] req@ffff8800a73cd880 x1818247332692992/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016844.343115:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332692992/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016844.343122:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016844.343125:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016844.343128:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332692992:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016844.343133:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332693120:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016844.343145:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016844.343151:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016844/real 1734016844] req@ffff8800a73cf480 x1818247332693120/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016844.343163:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332693120/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016844.343171:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016844.343173:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016844.343176:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332693120:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016844.396480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016845.398530:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016846.400479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016847.402492:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016848.404510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016849.350385:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016849.350395:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016849.350400:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016849.350410:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1111 00000100:00080000:3.0:1734016849.350421:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016849.350429:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016849.350467:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332693248/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016849.350504:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016849.350511:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016849.350513:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016849.350518:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1111 00000100:00080000:3.0:1734016849.350521:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016849.350525:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016849.350534:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016849.350545:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332693376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016849.350555:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016849.350559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016849.350560:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016849.350564:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1111 00000100:00080000:3.0:1734016849.350567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016849.350570:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016849.350575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016849.350621:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332693504/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016849.350631:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016849.350634:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016849.350636:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016849.350640:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1111 00000100:00080000:3.0:1734016849.350643:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016849.350646:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016849.350651:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016849.350660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332693632/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016849.350670:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016849.350673:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016849.350675:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016849.350679:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1111 00000100:00100000:1.0:1734016849.350681:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332693248:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016849.350681:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016849.350685:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016849.350689:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016849.350697:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332693760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016849.350705:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016849.350708:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016849.350710:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016849.350714:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1111 00000100:00080000:3.0:1734016849.350717:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016849.350721:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016849.350725:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016849.350733:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332693888/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016849.350741:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016849.350744:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016849.350746:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016849.350750:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1111 00000100:00080000:3.0:1734016849.350753:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016849.350756:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016849.350760:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016849.350770:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332694016/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734016849.350775:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016849.350805:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016849/real 1734016849] req@ffff8800a73cdf80 x1818247332693248/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016849.350828:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332693248/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016849.350841:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016849.350845:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016849.350850:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332693248:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734016849.350858:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332693376:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016849.350878:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016849.350885:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016849/real 1734016849] req@ffff8800a73cc700 x1818247332693376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016849.350896:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332693376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016849.350906:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016849.350909:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016849.350912:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332693376:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016849.350919:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332693504:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016849.350934:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016849.350940:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016849/real 1734016849] req@ffff8800a73cea00 x1818247332693504/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016849.350950:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332693504/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016849.350957:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016849.350960:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016849.350963:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332693504:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016849.350994:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332693632:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016849.351007:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016849.351014:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016849/real 1734016849] req@ffff8800a73cdc00 x1818247332693632/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016849.351025:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332693632/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016849.351033:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016849.351035:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016849.351038:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332693632:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016849.351046:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332693760:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016849.351059:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016849.351065:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016849/real 1734016849] req@ffff8800a73cf100 x1818247332693760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016849.351076:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332693760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016849.351084:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016849.351087:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016849.351090:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332693760:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016849.351095:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332693888:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016849.351108:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016849.351114:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016849/real 1734016849] req@ffff8800a73cce00 x1818247332693888/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016849.351124:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332693888/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016849.351130:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016849.351133:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016849.351136:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332693888:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016849.351141:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332694016:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016849.351155:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016849.351161:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016849/real 1734016849] req@ffff8800a73cc380 x1818247332694016/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016849.351172:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332694016/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016849.351180:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016849.351183:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016849.351185:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332694016:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016849.406446:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016850.408511:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016851.410496:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016852.412511:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016853.414480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016854.358361:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016854.358372:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016854.358377:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016854.358387:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1116 00000100:00080000:3.0:1734016854.358398:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016854.358406:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016854.358443:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332694144/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016854.358479:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016854.358483:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016854.358485:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016854.358490:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1116 00000100:00080000:3.0:1734016854.358537:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016854.358543:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016854.358553:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016854.358567:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332694272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016854.358580:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016854.358583:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016854.358585:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016854.358588:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1116 00000100:00080000:3.0:1734016854.358591:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016854.358594:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016854.358599:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016854.358607:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332694400/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016854.358617:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016854.358621:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016854.358622:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734016854.358625:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332694144:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016854.358626:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1116 00000100:00080000:3.0:1734016854.358628:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016854.358632:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016854.358636:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016854.358645:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332694528/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016854.358653:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016854.358656:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016854.358658:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016854.358661:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1116 00000100:00080000:3.0:1734016854.358665:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016854.358668:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016854.358672:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016854.358680:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332694656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016854.358688:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016854.358691:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016854.358693:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016854.358697:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1116 00000100:00080000:3.0:1734016854.358710:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016854.358713:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734016854.358717:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734016854.358718:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016854.358725:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332694784/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016854.358734:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016854.358737:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016854.358739:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016854.358742:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1116 00000100:00080000:3.0:1734016854.358745:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734016854.358746:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016854/real 1734016854] req@ffff8800a73ce680 x1818247332694144/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016854.358749:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016854.358753:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016854.358761:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332694912/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016854.358770:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332694144/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016854.358783:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016854.358787:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016854.358792:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332694144:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734016854.358801:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332694272:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016854.358820:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016854.358827:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016854/real 1734016854] req@ffff8800a73cc380 x1818247332694272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016854.358839:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332694272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016854.358849:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016854.358852:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016854.358855:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332694272:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016854.358884:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332694400:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016854.358899:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016854.358905:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016854/real 1734016854] req@ffff8800a73cce00 x1818247332694400/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016854.358916:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332694400/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016854.358924:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016854.358927:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016854.358930:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332694400:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016854.358936:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332694528:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016854.358948:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016854.358954:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016854/real 1734016854] req@ffff8800a73cf100 x1818247332694528/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016854.358964:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332694528/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016854.358971:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016854.358974:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016854.358977:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332694528:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016854.358984:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332694656:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016854.358998:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016854.359005:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016854/real 1734016854] req@ffff8800a73cdc00 x1818247332694656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016854.359015:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332694656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016854.359022:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016854.359024:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016854.359027:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332694656:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016854.359033:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332694784:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016854.359046:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016854.359052:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016854/real 1734016854] req@ffff8800a73cea00 x1818247332694784/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016854.359062:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332694784/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016854.359069:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016854.359071:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016854.359074:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332694784:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016854.359080:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332694912:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016854.359094:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016854.359100:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016854/real 1734016854] req@ffff8800a73cc700 x1818247332694912/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016854.359112:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332694912/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016854.359120:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016854.359122:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016854.359125:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332694912:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016854.416525:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016855.418529:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016856.420536:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016857.422539:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016858.424508:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016859.366342:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016859.366353:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016859.366358:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016859.366372:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1121 00000100:00080000:3.0:1734016859.366384:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016859.366394:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016859.366471:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332695040/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016859.366508:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016859.366513:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016859.366515:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016859.366520:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1121 00000100:00080000:3.0:1734016859.366524:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016859.366528:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016859.366540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016859.366552:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332695168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016859.366563:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016859.366567:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016859.366569:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016859.366573:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1121 00000100:00080000:3.0:1734016859.366576:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016859.366581:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016859.366586:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016859.366595:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332695296/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016859.366603:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016859.366606:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016859.366608:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016859.366611:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1121 00000100:00080000:3.0:1734016859.366614:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016859.366617:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016859.366622:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016859.366630:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332695424/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016859.366637:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332695040:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016859.366639:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016859.366643:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016859.366644:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016859.366648:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1121 00000100:00080000:3.0:1734016859.366651:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016859.366654:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016859.366659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016859.366667:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332695552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016859.366682:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016859.366686:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016859.366687:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016859.366692:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1121 00000100:00080000:3.0:1734016859.366695:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016859.366698:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016859.366703:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016859.366710:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332695680/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016859.366718:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016859.366721:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016859.366723:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734016859.366727:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016859.366727:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1121 00000100:00080000:3.0:1734016859.366730:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016859.366733:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016859.366738:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016859.366748:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332695808/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016859.366755:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016859/real 1734016859] req@ffff8800a73cdf80 x1818247332695040/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016859.366777:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332695040/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016859.366789:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016859.366793:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016859.366798:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332695040:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734016859.366805:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332695168:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016859.366824:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016859.366831:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016859/real 1734016859] req@ffff8800a73cf480 x1818247332695168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016859.366842:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332695168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016859.366852:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016859.366854:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016859.366858:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332695168:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016859.366864:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332695296:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016859.366879:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016859.366885:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016859/real 1734016859] req@ffff8800a73cd880 x1818247332695296/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016859.366895:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332695296/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016859.366902:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016859.366904:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016859.366908:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332695296:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016859.366937:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332695424:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016859.366950:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016859.366956:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016859/real 1734016859] req@ffff8800a73cfb80 x1818247332695424/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016859.366967:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332695424/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016859.366975:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016859.366977:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016859.366980:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332695424:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016859.366986:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332695552:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016859.366999:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016859.367005:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016859/real 1734016859] req@ffff8800a73cd500 x1818247332695552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016859.367016:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332695552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016859.367024:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016859.367026:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016859.367029:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332695552:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016859.367034:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332695680:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016859.367046:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016859.367052:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016859/real 1734016859] req@ffff8800a73cf800 x1818247332695680/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016859.367061:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332695680/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016859.367068:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016859.367070:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016859.367073:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332695680:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016859.367078:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332695808:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016859.367091:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016859.367097:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016859/real 1734016859] req@ffff8800a73ced80 x1818247332695808/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016859.367108:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332695808/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016859.367117:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016859.367119:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016859.367122:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332695808:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016859.426491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016860.428491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016861.430542:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016862.432516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016863.434467:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016864.374412:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016864.374424:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016864.374431:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016864.374445:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1126 00000100:00080000:3.0:1734016864.374458:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016864.374468:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016864.374518:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332695936/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016864.374555:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016864.374560:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016864.374562:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016864.374567:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1126 00000100:00080000:3.0:1734016864.374571:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016864.374575:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016864.374587:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016864.374602:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332696064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016864.374613:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016864.374616:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016864.374618:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016864.374623:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1126 00000100:00080000:3.0:1734016864.374626:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016864.374630:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016864.374635:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016864.374644:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332696192/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016864.374652:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016864.374656:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016864.374658:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016864.374662:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1126 00000100:00080000:3.0:1734016864.374665:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016864.374668:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016864.374673:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734016864.374679:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332695936:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734016864.374681:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332696320/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016864.374690:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016864.374694:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016864.374695:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016864.374699:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1126 00000100:00080000:3.0:1734016864.374701:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016864.374705:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016864.374709:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016864.374717:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332696448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016864.374725:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016864.374728:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016864.374730:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016864.374734:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1126 00000100:00080000:3.0:1734016864.374738:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016864.374741:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016864.374746:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016864.374754:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332696576/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016864.374763:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016864.374766:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1734016864.374767:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016864.374768:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016864.374771:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1126 00000100:00080000:3.0:1734016864.374774:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016864.374777:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016864.374782:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016864.374790:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332696704/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016864.374795:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016864/real 1734016864] req@ffff8800a73cca80 x1818247332695936/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016864.374817:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332695936/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016864.374830:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016864.374833:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016864.374838:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332695936:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734016864.374845:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332696064:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016864.374864:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016864.374870:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016864/real 1734016864] req@ffff8800a73ced80 x1818247332696064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016864.374882:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332696064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016864.374892:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016864.374894:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016864.374898:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332696064:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016864.374904:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332696192:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016864.374919:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016864.374925:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016864/real 1734016864] req@ffff8800a73cf800 x1818247332696192/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016864.374935:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332696192/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016864.374942:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016864.374945:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016864.374948:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332696192:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016864.374979:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332696320:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016864.374992:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016864.374998:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016864/real 1734016864] req@ffff8800a73cd500 x1818247332696320/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016864.375009:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332696320/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016864.375016:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016864.375018:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016864.375021:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332696320:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016864.375027:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332696448:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016864.375040:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016864.375047:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016864/real 1734016864] req@ffff8800a73cfb80 x1818247332696448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016864.375057:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332696448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016864.375065:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016864.375067:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016864.375070:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332696448:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016864.375075:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332696576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016864.375087:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016864.375092:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016864/real 1734016864] req@ffff8800a73cd880 x1818247332696576/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016864.375102:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332696576/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016864.375109:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016864.375111:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016864.375114:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332696576:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016864.375119:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332696704:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016864.375131:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016864.375137:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016864/real 1734016864] req@ffff8800a73cf480 x1818247332696704/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016864.375148:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332696704/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016864.375156:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016864.375158:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016864.375161:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332696704:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016864.436506:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016865.438475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016866.440529:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016867.442505:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016868.444525:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016869.382394:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016869.382405:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016869.382411:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016869.382426:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1131 00000100:00080000:3.0:1734016869.382438:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016869.382448:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016869.382498:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332696832/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016869.382537:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016869.382545:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016869.382547:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016869.382552:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1131 00000100:00080000:3.0:1734016869.382555:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016869.382559:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016869.382572:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016869.382583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332696960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016869.382593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016869.382597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016869.382599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016869.382603:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1131 00000100:00080000:3.0:1734016869.382606:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016869.382610:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016869.382614:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016869.382623:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332697088/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016869.382632:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016869.382635:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016869.382637:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016869.382641:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1131 00000100:00080000:3.0:1734016869.382643:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016869.382647:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016869.382653:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016869.382663:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332697216/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016869.382671:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332696832:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016869.382673:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016869.382676:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016869.382678:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016869.382682:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1131 00000100:00080000:3.0:1734016869.382684:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016869.382688:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016869.382692:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016869.382701:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332697344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016869.382710:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016869.382713:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016869.382715:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016869.382720:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1131 00000100:00080000:3.0:1734016869.382723:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016869.382726:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016869.382731:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016869.382739:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332697472/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016869.382746:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734016869.382748:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016869.382750:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016869.382752:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016869.382756:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1131 00000100:00080000:3.0:1734016869.382759:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016869.382763:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016869.382767:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734016869.382772:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016869/real 1734016869] req@ffff8800a73cdf80 x1818247332696832/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016869.382779:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332697600/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016869.382792:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332696832/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016869.382804:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016869.382807:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016869.382812:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332696832:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734016869.382821:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332696960:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016869.382839:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016869.382846:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016869/real 1734016869] req@ffff8800a73cc700 x1818247332696960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016869.382857:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332696960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016869.382867:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016869.382870:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016869.382873:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332696960:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016869.382880:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332697088:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016869.382894:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016869.382900:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016869/real 1734016869] req@ffff8800a73cea00 x1818247332697088/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016869.382910:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332697088/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016869.382917:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016869.382920:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016869.382924:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332697088:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016869.382950:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332697216:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016869.382963:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016869.382970:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016869/real 1734016869] req@ffff8800a73cdc00 x1818247332697216/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016869.382980:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332697216/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016869.382988:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016869.382991:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016869.382993:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332697216:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016869.383000:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332697344:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016869.383013:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016869.383019:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016869/real 1734016869] req@ffff8800a73cf100 x1818247332697344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016869.383029:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332697344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016869.383037:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016869.383039:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016869.383042:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332697344:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016869.383048:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332697472:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016869.383060:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016869.383066:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016869/real 1734016869] req@ffff8800a73cce00 x1818247332697472/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016869.383076:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332697472/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016869.383083:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016869.383086:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016869.383088:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332697472:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016869.383094:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332697600:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016869.383106:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016869.383112:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016869/real 1734016869] req@ffff8800a73cc380 x1818247332697600/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016869.383122:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332697600/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016869.383130:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016869.383133:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016869.383136:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332697600:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016869.446490:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016870.448494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016871.450434:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016872.452457:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016873.454472:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016874.390381:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016874.390393:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016874.390399:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016874.390413:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1136 00000100:00080000:3.0:1734016874.390425:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016874.390436:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016874.390486:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332697728/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016874.390527:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016874.390532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016874.390534:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016874.390539:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1136 00000100:00080000:3.0:1734016874.390543:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016874.390547:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016874.390560:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016874.390575:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332697856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016874.390586:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016874.390589:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016874.390591:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016874.390595:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1136 00000100:00080000:3.0:1734016874.390598:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016874.390601:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016874.390606:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016874.390616:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332697984/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016874.390624:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016874.390628:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016874.390629:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016874.390633:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1136 00000100:00080000:3.0:1734016874.390635:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016874.390639:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016874.390644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016874.390654:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332698112/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016874.390656:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332697728:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016874.390664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016874.390667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016874.390669:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016874.390673:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1136 00000100:00080000:3.0:1734016874.390676:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016874.390680:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016874.390684:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016874.390693:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332698240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016874.390702:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016874.390705:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016874.390707:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016874.390711:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1136 00000100:00080000:3.0:1734016874.390724:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734016874.390728:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016874.390728:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016874.390733:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016874.390741:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332698368/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016874.390750:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734016874.390753:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016874/real 1734016874] req@ffff8800a73ce680 x1818247332697728/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016874.390753:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016874.390754:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016874.390758:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1136 00000100:00080000:3.0:1734016874.390761:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016874.390765:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016874.390769:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734016874.390774:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332697728/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016874.390778:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332698496/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734016874.390787:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016874.390790:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016874.390795:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332697728:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734016874.390804:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332697856:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016874.390822:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016874.390829:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016874/real 1734016874] req@ffff8800a73cc380 x1818247332697856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016874.390840:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332697856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016874.390849:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016874.390852:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016874.390856:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332697856:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016874.390862:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332697984:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016874.390876:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016874.390882:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016874/real 1734016874] req@ffff8800a73cce00 x1818247332697984/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016874.390892:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332697984/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016874.390899:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016874.390902:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016874.390905:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332697984:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016874.390931:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332698112:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016874.390944:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016874.390950:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016874/real 1734016874] req@ffff8800a73cf100 x1818247332698112/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016874.390961:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332698112/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016874.390969:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016874.390971:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016874.390974:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332698112:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016874.390980:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332698240:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016874.390993:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016874.390999:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016874/real 1734016874] req@ffff8800a73cdc00 x1818247332698240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016874.391009:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332698240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016874.391017:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016874.391019:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016874.391022:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332698240:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016874.391028:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332698368:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016874.391040:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016874.391046:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016874/real 1734016874] req@ffff8800a73cea00 x1818247332698368/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016874.391056:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332698368/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016874.391063:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016874.391066:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016874.391069:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332698368:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016874.391075:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332698496:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016874.391087:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016874.391093:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016874/real 1734016874] req@ffff8800a73cc700 x1818247332698496/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016874.391104:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332698496/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016874.391111:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016874.391114:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016874.391117:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332698496:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016874.456521:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016875.458495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016876.460516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016877.462439:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016878.464534:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016879.398359:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016879.398371:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016879.398377:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016879.398391:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1141 00000100:00080000:3.0:1734016879.398402:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016879.398412:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016879.398460:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332698624/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016879.398498:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016879.398502:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016879.398504:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016879.398510:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1141 00000100:00080000:3.0:1734016879.398513:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016879.398517:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016879.398529:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016879.398541:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332698752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016879.398551:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016879.398555:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016879.398557:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016879.398561:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1141 00000100:00080000:3.0:1734016879.398564:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016879.398568:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016879.398573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016879.398582:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332698880/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016879.398591:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016879.398594:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016879.398596:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016879.398600:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1141 00000100:00080000:3.0:1734016879.398602:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016879.398606:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016879.398610:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016879.398619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332699008/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016879.398626:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332698624:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016879.398629:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016879.398633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016879.398634:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016879.398638:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1141 00000100:00080000:3.0:1734016879.398640:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016879.398644:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016879.398649:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016879.398657:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332699136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016879.398673:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016879.398676:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016879.398678:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016879.398682:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1141 00000100:00080000:3.0:1734016879.398685:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016879.398689:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016879.398693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016879.398701:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332699264/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734016879.398704:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016879.398709:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016879.398712:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016879.398714:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016879.398718:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1141 00000100:00080000:3.0:1734016879.398723:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016879.398726:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734016879.398728:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016879/real 1734016879] req@ffff8800a73cdf80 x1818247332698624/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734016879.398730:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016879.398741:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332699392/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016879.398748:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332698624/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016879.398760:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016879.398764:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016879.398769:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332698624:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734016879.398778:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332698752:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016879.398796:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016879.398803:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016879/real 1734016879] req@ffff8800a73cf480 x1818247332698752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016879.398814:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332698752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016879.398823:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016879.398826:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016879.398830:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332698752:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016879.398836:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332698880:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016879.398850:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016879.398857:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016879/real 1734016879] req@ffff8800a73cd880 x1818247332698880/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016879.398867:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332698880/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016879.398874:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016879.398877:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016879.398880:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332698880:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016879.398907:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332699008:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016879.398920:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016879.398926:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016879/real 1734016879] req@ffff8800a73cfb80 x1818247332699008/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016879.398937:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332699008/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016879.398944:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016879.398946:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016879.398949:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332699008:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016879.398956:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332699136:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734016879.398968:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016879.398974:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016879/real 1734016879] req@ffff8800a73cd500 x1818247332699136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016879.398985:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332699136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016879.398993:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016879.398995:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016879.398998:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332699136:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016879.399003:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332699264:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016879.399015:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016879.399021:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016879/real 1734016879] req@ffff8800a73cf800 x1818247332699264/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016879.399031:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332699264/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016879.399038:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016879.399040:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016879.399043:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332699264:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016879.399049:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332699392:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734016879.399062:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734016879.399115:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016879/real 1734016879] req@ffff8800a73ced80 x1818247332699392/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734016879.399130:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332699392/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734016879.399139:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734016879.399141:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734016879.399144:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332699392:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016879.466489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016880.468509:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016881.470512:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016882.472477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016883.474502:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016884.406364:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016884.406376:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016884.406382:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016884.406398:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1146 00000100:00080000:3.0:1734016884.406410:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016884.406420:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016884.406469:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332699520/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016884.406509:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016884.406513:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016884.406516:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016884.406521:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1146 00000100:00080000:3.0:1734016884.406524:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016884.406529:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016884.406542:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016884.406557:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332699648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016884.406568:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016884.406572:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016884.406573:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016884.406578:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1146 00000100:00080000:3.0:1734016884.406580:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016884.406584:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016884.406589:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016884.406598:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332699776/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016884.406610:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016884.406614:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016884.406616:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016884.406620:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1146 00000100:00080000:3.0:1734016884.406623:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016884.406626:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734016884.406630:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332699520:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734016884.406631:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016884.406641:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332699904/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016884.406649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016884.406652:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016884.406654:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016884.406657:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1146 00000100:00080000:3.0:1734016884.406661:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016884.406664:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016884.406669:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016884.406677:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332700032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016884.406685:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016884.406689:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016884.406690:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016884.406694:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1146 00000100:00080000:3.0:1734016884.406698:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016884.406701:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016884.406706:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016884.406714:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332700160/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016884.406723:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016884.406727:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016884.406728:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734016884.406730:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332699648:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734016884.406732:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1146 00000100:00080000:3.0:1734016884.406734:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016884.406738:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016884.406742:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016884.406752:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332700288/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734016884.406761:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332699776:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016884.406778:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332699904:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016884.406794:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332700032:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734016884.406809:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332700160:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734016884.406826:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332700288:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016884.476494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016885.478455:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016886.480489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734016887.412502:0:1956:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734016887.412551:0:1956:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734016887.412570:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734016887.412607:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734016887.412615:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734016887.412623:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734016887.412629:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734016887.412635:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734016887.412642:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:0.0:1734016887.412677:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016884/real 1734016887] req@ffff8800a73cf480 x1818247332700288/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016887.412701:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332700288/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016887.412713:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016887.412717:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016887.412723:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332700288:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016887.412730:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016884/real 1734016887] req@ffff8800a73cf800 x1818247332699776/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016887.412741:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332699776/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016887.412749:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016887.412752:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016887.412756:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332699776:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016887.412762:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016884/real 1734016887] req@ffff8800a73cd500 x1818247332699904/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016887.412774:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332699904/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016887.412786:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016887.412790:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016887.412795:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332699904:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016887.412804:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016884/real 1734016887] req@ffff8800a73cfb80 x1818247332700032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016887.412835:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332700032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016887.412848:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016887.412851:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016887.412857:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332700032:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016887.412867:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016884/real 1734016887] req@ffff8800a73cd880 x1818247332700160/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016887.412885:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332700160/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016887.412897:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016887.412901:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016887.412907:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332700160:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016887.412917:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016884/real 1734016887] req@ffff8800a73cca80 x1818247332699520/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016887.412996:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332699520/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016887.413010:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016887.413014:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016887.413019:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332699520:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016887.413029:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016884/real 1734016887] req@ffff8800a73ced80 x1818247332699648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016887.413047:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332699648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016887.413059:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016887.413062:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016887.413068:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332699648:192.168.201.152@tcp:38:kworker.0 00010000:00010000:1.0:1734016887.482468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016888.484542:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016889.414387:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016889.414398:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016889.414404:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016889.414418:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1151 00000100:00080000:3.0:1734016889.414431:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016889.414441:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016889.414492:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332700416/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016889.414531:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016889.414538:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016889.414541:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016889.414546:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1151 00000100:00080000:3.0:1734016889.414550:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016889.414554:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016889.414567:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016889.414578:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332700544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016889.414589:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016889.414592:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016889.414594:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016889.414598:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1151 00000100:00080000:3.0:1734016889.414601:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016889.414605:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016889.414611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016889.414620:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332700672/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016889.414628:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016889.414631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016889.414633:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016889.414637:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1151 00000100:00080000:3.0:1734016889.414639:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016889.414643:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016889.414648:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016889.414657:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332700800/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016889.414659:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332700416:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016889.414667:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016889.414671:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016889.414672:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016889.414676:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1151 00000100:00080000:3.0:1734016889.414678:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016889.414682:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016889.414687:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016889.414696:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332700928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016889.414704:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016889.414707:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016889.414709:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016889.414713:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1151 00000100:00080000:3.0:1734016889.414716:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016889.414720:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016889.414725:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734016889.414731:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734016889.414733:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332701056/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016889.414743:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016889.414746:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016889.414748:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016889.414752:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1151 00000100:00100000:0.0:1734016889.414754:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016889/real 1734016889] req@ffff8800a73cdf80 x1818247332700416/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016889.414755:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016889.414759:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016889.414764:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016889.414773:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332700416/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016889.414775:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332701184/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016889.414785:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016889.414789:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016889.414794:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332700416:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016889.414803:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332700544:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016889.414822:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016889.414829:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016889/real 1734016889] req@ffff8800a73cc700 x1818247332700544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016889.414840:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332700544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016889.414848:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016889.414851:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016889.414854:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332700544:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016889.414861:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332700672:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016889.414877:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016889.414883:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016889/real 1734016889] req@ffff8800a73cea00 x1818247332700672/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016889.414893:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332700672/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016889.414950:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016889.414952:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016889.414956:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332700672:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016889.414982:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332700800:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016889.414999:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016889.415007:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016889/real 1734016889] req@ffff8800a73cdc00 x1818247332700800/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016889.415018:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332700800/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016889.415026:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016889.415028:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016889.415031:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332700800:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016889.415037:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332700928:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016889.415050:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016889.415055:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016889/real 1734016889] req@ffff8800a73cf100 x1818247332700928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016889.415065:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332700928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016889.415073:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016889.415075:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016889.415078:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332700928:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016889.415084:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332701056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016889.415096:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016889.415102:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016889/real 1734016889] req@ffff8800a73cce00 x1818247332701056/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016889.415112:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332701056/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016889.415127:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016889.415130:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016889.415133:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332701056:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016889.415139:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332701184:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016889.415151:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016889.415157:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016889/real 1734016889] req@ffff8800a73cc380 x1818247332701184/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016889.415167:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332701184/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016889.415174:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016889.415177:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016889.415180:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332701184:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016889.486487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016890.488521:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016891.490506:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016892.492501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016893.494481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016894.422367:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016894.422378:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016894.422384:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016894.422398:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1156 00000100:00080000:3.0:1734016894.422409:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016894.422419:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016894.422469:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332701312/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016894.422505:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016894.422510:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016894.422512:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016894.422517:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1156 00000100:00080000:3.0:1734016894.422521:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016894.422525:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016894.422537:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016894.422551:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332701440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016894.422563:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016894.422567:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016894.422569:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016894.422572:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1156 00000100:00080000:3.0:1734016894.422575:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016894.422579:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016894.422584:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016894.422593:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332701568/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016894.422601:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016894.422604:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016894.422606:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016894.422609:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1156 00000100:00080000:3.0:1734016894.422612:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016894.422616:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734016894.422620:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332701312:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734016894.422620:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016894.422629:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332701696/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016894.422638:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016894.422641:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016894.422643:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016894.422647:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1156 00000100:00080000:3.0:1734016894.422650:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016894.422654:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016894.422658:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016894.422667:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332701824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016894.422675:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016894.422678:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016894.422680:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016894.422684:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1156 00000100:00080000:3.0:1734016894.422687:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016894.422690:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016894.422695:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016894.422702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332701952/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734016894.422705:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016894.422710:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016894.422714:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016894.422715:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016894.422719:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1156 00000100:00080000:3.0:1734016894.422722:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016894.422725:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016894.422730:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016894.422733:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016894/real 1734016894] req@ffff8800a73ce680 x1818247332701312/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016894.422739:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332702080/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016894.422755:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332701312/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016894.422767:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016894.422770:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016894.422775:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332701312:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016894.422785:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332701440:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016894.422805:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016894.422852:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016894/real 1734016894] req@ffff8800a73cc380 x1818247332701440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016894.422867:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332701440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016894.422876:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016894.422879:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016894.422883:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332701440:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016894.422889:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332701568:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016894.422908:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016894.422915:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016894/real 1734016894] req@ffff8800a73cce00 x1818247332701568/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016894.422925:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332701568/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016894.422934:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016894.422936:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016894.422939:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332701568:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016894.422967:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332701696:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016894.422980:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016894.422986:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016894/real 1734016894] req@ffff8800a73cf100 x1818247332701696/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016894.422998:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332701696/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016894.423006:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016894.423008:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016894.423011:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332701696:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016894.423015:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332701824:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016894.423027:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016894.423033:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016894/real 1734016894] req@ffff8800a73cdc00 x1818247332701824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016894.423042:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332701824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016894.423050:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016894.423052:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016894.423055:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332701824:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016894.423061:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332701952:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016894.423073:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016894.423079:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016894/real 1734016894] req@ffff8800a73cea00 x1818247332701952/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016894.423089:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332701952/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016894.423096:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016894.423099:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016894.423102:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332701952:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016894.423108:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332702080:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016894.423121:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016894.423127:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016894/real 1734016894] req@ffff8800a73cc700 x1818247332702080/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016894.423136:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332702080/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016894.423144:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016894.423146:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016894.423149:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332702080:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016894.496490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016895.498521:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016896.500517:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016897.502423:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016898.504453:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016899.430341:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016899.430352:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016899.430358:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016899.430372:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1161 00000100:00080000:3.0:1734016899.430384:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016899.430394:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016899.430442:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332702208/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016899.430481:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016899.430485:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016899.430487:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016899.430492:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1161 00000100:00080000:3.0:1734016899.430496:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016899.430500:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016899.430512:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016899.430524:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332702336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016899.430534:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016899.430538:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016899.430540:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016899.430544:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1161 00000100:00080000:3.0:1734016899.430548:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016899.430552:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016899.430557:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016899.430566:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332702464/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016899.430576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016899.430579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016899.430581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016899.430585:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1161 00000100:00080000:3.0:1734016899.430588:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016899.430592:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734016899.430595:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332702208:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734016899.430597:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016899.430605:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332702592/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016899.430613:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016899.430617:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016899.430619:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016899.430623:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1161 00000100:00080000:3.0:1734016899.430627:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016899.430630:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016899.430635:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016899.430644:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332702720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016899.430659:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016899.430662:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016899.430664:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016899.430668:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1161 00000800:00000100:0.0:1734016899.430669:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016899.430672:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016899.430675:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016899.430679:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016899.430692:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016899/real 1734016899] req@ffff8800a73cdf80 x1818247332702208/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016899.430712:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332702208/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016899.430724:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016899.430771:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734016899.430773:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332702848/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016899.430777:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332702208:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016899.430785:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734016899.430786:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332702336:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734016899.430789:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016899.430790:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016899.430795:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1161 00000100:00080000:3.0:1734016899.430798:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016899.430802:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016899.430808:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734016899.430812:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016899.430821:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016899/real 1734016899] req@ffff8800a73ced80 x1818247332702336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016899.430822:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332702976/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016899.430833:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332702336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016899.430842:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016899.430845:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016899.430849:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332702336:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016899.430875:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332702464:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016899.430889:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016899.430896:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016899/real 1734016899] req@ffff8800a73cca80 x1818247332702464/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016899.430906:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332702464/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016899.430914:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016899.430916:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016899.430919:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332702464:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016899.430925:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332702592:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016899.430937:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016899.430943:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016899/real 1734016899] req@ffff8800a73cd880 x1818247332702592/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016899.430960:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332702592/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016899.430968:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016899.430970:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016899.430973:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332702592:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016899.430978:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332702720:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016899.430992:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016899.430997:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016899/real 1734016899] req@ffff8800a73cfb80 x1818247332702720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016899.431007:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332702720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016899.431015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016899.431017:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016899.431020:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332702720:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016899.431026:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332702848:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016899.431038:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016899.431044:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016899/real 1734016899] req@ffff8800a73cd500 x1818247332702848/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016899.431054:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332702848/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016899.431061:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016899.431064:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016899.431067:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332702848:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016899.431074:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332702976:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016899.431087:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016899.431092:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016899/real 1734016899] req@ffff8800a73cf800 x1818247332702976/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016899.431102:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332702976/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016899.431109:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016899.431112:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016899.431115:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332702976:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016899.506510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016900.508510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016901.510495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016902.512473:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016903.514456:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016904.438399:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016904.438411:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016904.438417:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016904.438431:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1166 00000100:00080000:3.0:1734016904.438444:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016904.438454:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016904.438504:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332703104/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016904.438542:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016904.438547:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016904.438549:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016904.438555:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1166 00000100:00080000:3.0:1734016904.438558:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016904.438562:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016904.438575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016904.438589:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332703232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016904.438599:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016904.438603:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016904.438605:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016904.438609:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1166 00000100:00080000:3.0:1734016904.438612:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016904.438615:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016904.438621:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016904.438630:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332703360/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016904.438638:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016904.438676:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016904.438678:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016904.438682:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1166 00000100:00080000:3.0:1734016904.438686:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016904.438689:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016904.438695:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016904.438705:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332703488/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016904.438709:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332703104:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016904.438716:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016904.438720:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016904.438721:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016904.438726:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1166 00000100:00080000:3.0:1734016904.438728:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016904.438732:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016904.438737:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016904.438745:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332703616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016904.438754:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016904.438757:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016904.438759:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016904.438764:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1166 00000100:00080000:3.0:1734016904.438768:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016904.438772:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016904.438777:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734016904.438784:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734016904.438786:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332703744/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016904.438794:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016904.438798:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016904.438799:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016904.438803:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1166 00000100:00080000:3.0:1734016904.438806:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734016904.438807:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016904/real 1734016904] req@ffff8800a73cf480 x1818247332703104/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016904.438810:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016904.438814:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016904.438823:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332703872/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016904.438827:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332703104/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016904.438839:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016904.438842:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016904.438847:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332703104:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016904.438856:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332703232:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016904.438878:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016904.438885:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016904/real 1734016904] req@ffff8800a73cf800 x1818247332703232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016904.438897:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332703232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016904.438904:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016904.438907:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016904.438911:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332703232:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016904.438917:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332703360:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016904.438931:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016904.438937:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016904/real 1734016904] req@ffff8800a73cd500 x1818247332703360/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016904.438947:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332703360/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016904.438954:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016904.438957:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016904.438960:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332703360:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016904.438986:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332703488:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016904.438998:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016904.439004:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016904/real 1734016904] req@ffff8800a73cfb80 x1818247332703488/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016904.439021:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332703488/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016904.439029:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016904.439031:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016904.439034:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332703488:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016904.439040:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332703616:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016904.439052:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016904.439058:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016904/real 1734016904] req@ffff8800a73cd880 x1818247332703616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016904.439068:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332703616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016904.439075:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016904.439078:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016904.439081:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332703616:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016904.439087:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332703744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016904.439099:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016904.439106:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016904/real 1734016904] req@ffff8800a73cca80 x1818247332703744/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016904.439115:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332703744/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016904.439122:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016904.439126:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016904.439129:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332703744:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016904.439135:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332703872:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016904.439148:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016904.439153:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016904/real 1734016904] req@ffff8800a73ced80 x1818247332703872/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016904.439163:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332703872/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016904.439170:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016904.439173:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016904.439176:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332703872:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016904.516496:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016905.518466:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016906.520480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016907.522498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016908.524481:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016909.446401:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016909.446412:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016909.446418:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016909.446433:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1171 00000100:00080000:3.0:1734016909.446445:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016909.446455:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016909.446506:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332704000/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016909.446546:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016909.446553:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016909.446555:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016909.446601:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1171 00000100:00080000:3.0:1734016909.446605:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016909.446610:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016909.446622:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016909.446635:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332704128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016909.446647:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016909.446651:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016909.446652:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016909.446656:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1171 00000100:00080000:3.0:1734016909.446659:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016909.446663:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016909.446668:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734016909.446671:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332704000:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734016909.446678:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332704256/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016909.446688:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016909.446691:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016909.446693:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016909.446697:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1171 00000100:00080000:3.0:1734016909.446700:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016909.446704:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016909.446710:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016909.446718:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332704384/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016909.446727:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016909.446731:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016909.446732:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016909.446736:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1171 00000100:00080000:3.0:1734016909.446739:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734016909.446743:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016909.446743:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016909.446748:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016909.446757:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332704512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016909.446765:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734016909.446767:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016909/real 1734016909] req@ffff8800a73cdf80 x1818247332704000/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016909.446768:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016909.446770:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016909.446775:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1171 00000100:00080000:3.0:1734016909.446778:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016909.446782:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016909.446786:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016909.446788:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332704000/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016909.446795:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332704640/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016909.446799:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016909.446803:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734016909.446803:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016909.446807:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734016909.446808:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332704000:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016909.446809:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016909.446813:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1171 00000100:00080000:3.0:1734016909.446815:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734016909.446816:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332704128:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734016909.446819:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016909.446824:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016909.446836:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332704768/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734016909.446838:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016909.446845:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016909/real 1734016909] req@ffff8800a73cc700 x1818247332704128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016909.446857:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332704128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016909.446865:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016909.446868:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016909.446872:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332704128:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016909.446897:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332704256:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016909.446911:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016909.446917:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016909/real 1734016909] req@ffff8800a73cea00 x1818247332704256/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016909.446927:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332704256/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016909.446935:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016909.446937:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016909.446940:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332704256:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016909.446946:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332704384:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016909.446958:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016909.446964:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016909/real 1734016909] req@ffff8800a73cdc00 x1818247332704384/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016909.446982:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332704384/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016909.446990:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016909.446993:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016909.446996:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332704384:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016909.447001:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332704512:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016909.447014:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016909.447019:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016909/real 1734016909] req@ffff8800a73cf100 x1818247332704512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016909.447029:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332704512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016909.447037:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016909.447039:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016909.447042:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332704512:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016909.447048:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332704640:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016909.447061:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016909.447067:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016909/real 1734016909] req@ffff8800a73cce00 x1818247332704640/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016909.447077:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332704640/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016909.447084:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016909.447087:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016909.447090:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332704640:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016909.447096:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332704768:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016909.447108:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016909.447114:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016909/real 1734016909] req@ffff8800a73cc380 x1818247332704768/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016909.447124:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332704768/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016909.447131:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016909.447133:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016909.447136:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332704768:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016909.526475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016910.528511:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016911.530467:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016912.532497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016913.534485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016914.454381:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016914.454392:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016914.454398:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016914.454412:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1176 00000100:00080000:3.0:1734016914.454423:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016914.454433:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016914.454525:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332704896/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016914.454563:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016914.454568:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016914.454570:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016914.454575:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1176 00000100:00080000:3.0:1734016914.454579:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016914.454584:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016914.454596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016914.454611:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332705024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016914.454623:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016914.454627:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016914.454629:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016914.454633:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1176 00000100:00080000:3.0:1734016914.454635:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016914.454639:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016914.454644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016914.454653:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332705152/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016914.454662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016914.454665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016914.454666:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016914.454670:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1176 00000100:00080000:3.0:1734016914.454673:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016914.454676:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016914.454681:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016914.454690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332705280/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016914.454695:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332704896:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016914.454699:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016914.454703:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016914.454704:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016914.454709:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1176 00000100:00080000:3.0:1734016914.454712:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016914.454716:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016914.454721:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016914.454730:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332705408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016914.454738:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016914.454741:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016914.454743:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016914.454747:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1176 00000100:00080000:3.0:1734016914.454750:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016914.454754:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016914.454758:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016914.454766:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332705536/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734016914.454770:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016914.454774:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016914.454777:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016914.454778:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016914.454782:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1176 00000100:00080000:3.0:1734016914.454787:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016914.454790:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734016914.454793:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016914/real 1734016914] req@ffff8800a73ce680 x1818247332704896/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734016914.454795:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016914.454805:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332705664/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016914.454813:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332704896/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016914.454825:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016914.454828:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016914.454833:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332704896:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016914.454842:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332705024:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016914.454863:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016914.454870:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016914/real 1734016914] req@ffff8800a73cc380 x1818247332705024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016914.454882:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332705024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016914.454890:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016914.454892:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016914.454896:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332705024:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016914.454902:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332705152:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016914.454916:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016914.454922:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016914/real 1734016914] req@ffff8800a73cce00 x1818247332705152/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016914.454932:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332705152/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016914.454939:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016914.454942:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016914.454945:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332705152:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016914.454971:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332705280:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016914.454984:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016914.454990:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016914/real 1734016914] req@ffff8800a73cf100 x1818247332705280/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016914.455007:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332705280/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016914.455015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016914.455018:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016914.455021:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332705280:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016914.455027:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332705408:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016914.455039:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016914.455044:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016914/real 1734016914] req@ffff8800a73cdc00 x1818247332705408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016914.455054:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332705408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016914.455062:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016914.455064:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016914.455067:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332705408:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016914.455073:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332705536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016914.455085:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016914.455091:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016914/real 1734016914] req@ffff8800a73cea00 x1818247332705536/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016914.455101:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332705536/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016914.455108:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016914.455112:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016914.455115:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332705536:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016914.455121:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332705664:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016914.455134:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016914.455140:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016914/real 1734016914] req@ffff8800a73cc700 x1818247332705664/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016914.455150:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332705664/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016914.455157:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016914.455159:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016914.455162:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332705664:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016914.536515:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016915.538527:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016916.540500:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016917.542465:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016918.544498:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016919.462349:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016919.462361:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016919.462366:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016919.462380:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1182 00000100:00080000:3.0:1734016919.462428:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016919.462439:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016919.462487:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332705792/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016919.462521:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016919.462525:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016919.462527:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016919.462532:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1182 00000100:00080000:3.0:1734016919.462536:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016919.462540:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016919.462552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016919.462564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332705920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016919.462574:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016919.462578:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016919.462580:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016919.462584:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1182 00000100:00080000:3.0:1734016919.462588:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016919.462592:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016919.462597:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016919.462606:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332706048/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016919.462615:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016919.462619:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016919.462620:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016919.462624:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1182 00000100:00080000:3.0:1734016919.462627:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734016919.462630:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332705792:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016919.462631:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016919.462635:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016919.462644:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332706176/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016919.462652:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016919.462656:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016919.462657:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016919.462661:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1182 00000100:00080000:3.0:1734016919.462663:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016919.462667:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016919.462671:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016919.462679:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332706304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016919.462694:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016919.462698:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016919.462699:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016919.462704:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1182 00000100:00080000:3.0:1734016919.462707:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016919.462710:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016919.462714:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734016919.462716:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734016919.462722:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332706432/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016919.462730:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016919.462734:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016919.462735:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016919.462739:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1182 00000100:00080000:3.0:1734016919.462741:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734016919.462742:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016919/real 1734016919] req@ffff8800a73cdf80 x1818247332705792/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016919.462745:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016919.462750:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016919.462759:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332706560/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016919.462764:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332705792/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016919.462776:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016919.462779:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016919.462784:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332705792:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016919.462791:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332705920:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016919.462814:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016919.462821:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016919/real 1734016919] req@ffff8800a73ced80 x1818247332705920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016919.462833:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332705920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016919.462841:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016919.462843:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016919.462847:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332705920:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016919.462875:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332706048:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016919.462889:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016919.462895:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016919/real 1734016919] req@ffff8800a73cca80 x1818247332706048/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016919.462906:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332706048/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016919.462913:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016919.462916:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016919.462919:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332706048:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016919.462925:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332706176:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016919.462936:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016919.462942:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016919/real 1734016919] req@ffff8800a73cd880 x1818247332706176/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016919.462959:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332706176/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016919.462967:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016919.462969:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016919.462972:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332706176:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016919.462977:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332706304:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016919.462989:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016919.462995:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016919/real 1734016919] req@ffff8800a73cfb80 x1818247332706304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016919.463005:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332706304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016919.463012:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016919.463014:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016919.463017:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332706304:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016919.463022:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332706432:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016919.463035:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016919.463041:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016919/real 1734016919] req@ffff8800a73cd500 x1818247332706432/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016919.463052:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332706432/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016919.463059:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016919.463062:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016919.463066:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332706432:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016919.463071:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332706560:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016919.463084:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016919.463090:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016919/real 1734016919] req@ffff8800a73cf800 x1818247332706560/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016919.463100:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332706560/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016919.463107:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016919.463110:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016919.463113:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332706560:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016919.546512:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016920.548478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016921.550522:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016922.552542:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016923.554511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016924.470407:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016924.470420:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016924.470425:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016924.470439:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1187 00000100:00080000:3.0:1734016924.470450:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016924.470460:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016924.470506:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332706688/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016924.470542:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016924.470547:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016924.470549:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016924.470554:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1187 00000100:00080000:3.0:1734016924.470558:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016924.470562:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016924.470574:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016924.470587:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332706816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016924.470598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016924.470601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016924.470603:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016924.470608:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1187 00000100:00080000:3.0:1734016924.470611:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016924.470614:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016924.470619:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016924.470628:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332706944/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016924.470638:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016924.470642:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016924.470643:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016924.470647:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1187 00000100:00080000:3.0:1734016924.470649:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734016924.470650:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332706688:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016924.470653:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016924.470657:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016924.470665:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332707072/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016924.470673:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016924.470677:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016924.470678:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016924.470682:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1187 00000100:00080000:3.0:1734016924.470684:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016924.470688:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016924.470692:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016924.470699:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332707200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016924.470708:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016924.470711:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016924.470713:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016924.470717:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1187 00000100:00080000:3.0:1734016924.470721:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016924.470724:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016924.470730:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734016924.470734:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734016924.470739:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332707328/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016924.470747:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016924.470751:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016924.470752:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016924.470756:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1187 00000100:00100000:0.0:1734016924.470759:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016924/real 1734016924] req@ffff8800a73cf480 x1818247332706688/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016924.470759:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016924.470762:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016924.470767:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016924.470776:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332707456/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016924.470781:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332706688/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016924.470793:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016924.470797:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016924.470802:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332706688:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016924.470809:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332706816:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016924.470832:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016924.470839:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016924/real 1734016924] req@ffff8800a73cf800 x1818247332706816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016924.470851:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332706816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016924.470859:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016924.470862:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016924.470865:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332706816:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016924.470892:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332706944:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016924.470908:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016924.470914:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016924/real 1734016924] req@ffff8800a73cd500 x1818247332706944/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016924.470924:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332706944/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016924.470931:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016924.470934:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016924.470937:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332706944:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016924.470943:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332707072:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016924.470954:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016924.470960:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016924/real 1734016924] req@ffff8800a73cfb80 x1818247332707072/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016924.470977:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332707072/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016924.470985:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016924.470988:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016924.470991:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332707072:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016924.470995:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332707200:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016924.471008:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016924.471013:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016924/real 1734016924] req@ffff8800a73cd880 x1818247332707200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016924.471023:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332707200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016924.471030:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016924.471032:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016924.471035:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332707200:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016924.471041:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332707328:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016924.471054:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016924.471060:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016924/real 1734016924] req@ffff8800a73cca80 x1818247332707328/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016924.471070:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332707328/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016924.471077:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016924.471080:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016924.471084:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332707328:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016924.471089:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332707456:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016924.471103:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016924.471108:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016924/real 1734016924] req@ffff8800a73ced80 x1818247332707456/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016924.471118:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332707456/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016924.471125:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016924.471127:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016924.471130:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332707456:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016924.556512:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016925.558422:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016926.560509:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016927.562531:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016928.564473:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016929.478362:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016929.478372:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016929.478378:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016929.478391:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1192 00000100:00080000:3.0:1734016929.478402:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016929.478412:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016929.478460:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332707584/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016929.478497:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016929.478504:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016929.478506:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016929.478511:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1192 00000100:00080000:3.0:1734016929.478515:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016929.478519:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016929.478531:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016929.478542:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332707712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016929.478553:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016929.478556:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016929.478558:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016929.478562:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1192 00000100:00080000:3.0:1734016929.478565:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016929.478569:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016929.478574:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016929.478582:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332707840/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016929.478592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016929.478595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016929.478596:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016929.478600:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1192 00000100:00080000:3.0:1734016929.478603:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016929.478606:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734016929.478607:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332707584:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734016929.478612:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016929.478619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332707968/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016929.478627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016929.478631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016929.478632:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016929.478635:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1192 00000100:00080000:3.0:1734016929.478638:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016929.478641:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016929.478646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016929.478653:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332708096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016929.478662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016929.478664:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016929.478666:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016929.478670:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1192 00000100:00080000:3.0:1734016929.478673:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016929.478676:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016929.478681:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016929.478688:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332708224/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734016929.478691:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016929.478696:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016929.478699:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016929.478701:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016929.478706:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1192 00000100:00100000:0.0:1734016929.478717:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016929/real 1734016929] req@ffff8800a73cdf80 x1818247332707584/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016929.478718:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016929.478722:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016929.478727:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016929.478738:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332708352/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016929.478739:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332707584/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016929.478751:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016929.478755:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016929.478760:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332707584:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016929.478767:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332707712:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016929.478789:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016929.478796:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016929/real 1734016929] req@ffff8800a73cc700 x1818247332707712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016929.478807:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332707712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016929.478815:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016929.478818:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016929.478821:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332707712:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016929.478849:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332707840:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016929.478863:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016929.478869:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016929/real 1734016929] req@ffff8800a73cea00 x1818247332707840/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016929.478879:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332707840/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016929.478887:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016929.478889:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016929.478892:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332707840:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016929.478898:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332707968:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016929.478909:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016929.478915:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016929/real 1734016929] req@ffff8800a73cdc00 x1818247332707968/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016929.478932:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332707968/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016929.478940:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016929.478942:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016929.478945:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332707968:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016929.478950:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332708096:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016929.478962:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016929.478968:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016929/real 1734016929] req@ffff8800a73cf100 x1818247332708096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016929.478978:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332708096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016929.478985:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016929.478987:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016929.478990:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332708096:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016929.478995:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332708224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016929.479008:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016929.479014:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016929/real 1734016929] req@ffff8800a73cce00 x1818247332708224/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016929.479023:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332708224/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016929.479030:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016929.479034:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016929.479037:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332708224:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016929.479043:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332708352:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016929.479056:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016929.479061:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016929/real 1734016929] req@ffff8800a73cc380 x1818247332708352/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016929.479071:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332708352/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016929.479078:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016929.479080:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016929.479083:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332708352:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016929.566510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016930.568516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016931.570500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016932.572411:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016933.574468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016934.486347:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016934.486358:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016934.486363:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016934.486373:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1197 00000100:00080000:3.0:1734016934.486383:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016934.486391:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016934.486427:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332708480/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016934.486460:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016934.486465:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016934.486467:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016934.486471:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1197 00000100:00080000:3.0:1734016934.486474:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016934.486478:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016934.486489:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016934.486501:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332708608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016934.486512:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016934.486515:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016934.486517:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016934.486521:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1197 00000100:00080000:3.0:1734016934.486523:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016934.486526:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016934.486531:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016934.486539:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332708736/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016934.486548:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016934.486551:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016934.486552:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016934.486556:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1197 00000100:00080000:3.0:1734016934.486558:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016934.486561:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016934.486566:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016934.486571:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332708480:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734016934.486574:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332708864/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016934.486583:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016934.486587:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016934.486588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016934.486591:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1197 00000100:00080000:3.0:1734016934.486594:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016934.486598:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016934.486602:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016934.486610:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332708992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016934.486618:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016934.486621:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016934.486622:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016934.486626:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1197 00000100:00080000:3.0:1734016934.486629:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016934.486632:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016934.486636:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734016934.486638:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734016934.486643:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332709120/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016934.486651:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016934.486654:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016934.486655:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734016934.486658:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016934/real 1734016934] req@ffff8800a73ce680 x1818247332708480/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016934.486659:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1197 00000100:00080000:3.0:1734016934.486661:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016934.486664:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016934.486669:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016934.486676:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332709248/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016934.486678:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332708480/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016934.486688:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016934.486691:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016934.486696:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332708480:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016934.486703:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332708608:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016934.486724:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016934.486730:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016934/real 1734016934] req@ffff8800a73cc380 x1818247332708608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016934.486742:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332708608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016934.486749:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016934.486752:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016934.486755:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332708608:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016934.486761:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332708736:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016934.486774:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016934.486780:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016934/real 1734016934] req@ffff8800a73cce00 x1818247332708736/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016934.486790:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332708736/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016934.486798:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016934.486800:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016934.486803:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332708736:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016934.486827:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332708864:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016934.486840:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016934.486846:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016934/real 1734016934] req@ffff8800a73cf100 x1818247332708864/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016934.486863:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332708864/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016934.486871:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016934.486873:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016934.486876:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332708864:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016934.486881:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332708992:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016934.486892:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016934.486898:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016934/real 1734016934] req@ffff8800a73cdc00 x1818247332708992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016934.486908:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332708992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016934.486915:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016934.486917:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016934.486920:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332708992:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016934.486925:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332709120:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016934.486937:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016934.486943:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016934/real 1734016934] req@ffff8800a73cea00 x1818247332709120/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016934.486952:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332709120/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016934.486959:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016934.486962:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016934.486965:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332709120:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016934.486970:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332709248:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016934.486982:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016934.486988:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016934/real 1734016934] req@ffff8800a73cc700 x1818247332709248/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016934.486998:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332709248/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016934.487005:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016934.487008:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016934.487011:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332709248:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016934.576457:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016935.578491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016936.580497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016937.582451:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016938.584442:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016939.494324:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016939.494334:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016939.494338:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016939.494349:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1202 00000100:00080000:3.0:1734016939.494358:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016939.494365:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016939.494400:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332709376/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016939.494433:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016939.494437:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016939.494439:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016939.494444:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1202 00000100:00080000:3.0:1734016939.494447:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016939.494451:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016939.494460:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016939.494470:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332709504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016939.494480:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016939.494484:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016939.494485:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016939.494489:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1202 00000100:00080000:3.0:1734016939.494492:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016939.494496:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016939.494500:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016939.494508:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332709632/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016939.494517:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016939.494520:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016939.494521:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016939.494525:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1202 00000100:00080000:3.0:1734016939.494527:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016939.494530:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016939.494535:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016939.494542:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332709376:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734016939.494542:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332709760/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016939.494551:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016939.494555:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016939.494556:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016939.494560:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1202 00000100:00080000:3.0:1734016939.494562:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016939.494565:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016939.494569:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016939.494577:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332709888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016939.494592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016939.494595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016939.494596:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016939.494600:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1202 00000100:00080000:3.0:1734016939.494603:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734016939.494605:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016939.494606:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016939.494611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016939.494618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332710016/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016939.494626:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734016939.494627:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016939/real 1734016939] req@ffff8800a73cdf80 x1818247332709376/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016939.494629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016939.494630:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016939.494634:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1202 00000100:00080000:3.0:1734016939.494636:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016939.494639:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016939.494643:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016939.494645:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332709376/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016939.494652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332710144/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016939.494656:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016939.494659:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016939.494664:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332709376:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016939.494671:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332709504:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016939.494691:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016939.494698:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016939/real 1734016939] req@ffff8800a73ced80 x1818247332709504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016939.494709:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332709504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016939.494717:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016939.494719:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016939.494722:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332709504:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016939.494728:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332709632:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016939.494741:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016939.494747:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016939/real 1734016939] req@ffff8800a73cca80 x1818247332709632/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016939.494757:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332709632/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016939.494764:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016939.494766:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016939.494769:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332709632:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016939.494794:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332709760:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016939.494807:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016939.494813:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016939/real 1734016939] req@ffff8800a73cd880 x1818247332709760/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016939.494830:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332709760/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016939.494838:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016939.494840:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016939.494843:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332709760:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016939.494848:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332709888:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016939.494860:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016939.494866:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016939/real 1734016939] req@ffff8800a73cfb80 x1818247332709888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016939.494875:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332709888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016939.494883:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016939.494885:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016939.494888:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332709888:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016939.494894:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332710016:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016939.494905:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016939.494911:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016939/real 1734016939] req@ffff8800a73cd500 x1818247332710016/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016939.494921:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332710016/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016939.494928:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016939.494931:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016939.494934:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332710016:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016939.494940:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332710144:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016939.494953:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016939.494958:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016939/real 1734016939] req@ffff8800a73cf800 x1818247332710144/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016939.494968:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332710144/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016939.494976:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016939.494978:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016939.494981:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332710144:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016939.586504:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016940.588477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016941.590493:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016942.592452:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016943.594481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016944.502289:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016944.502299:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016944.502303:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016944.502314:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1207 00000100:00080000:3.0:1734016944.502324:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016944.502332:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016944.502369:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332710272/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016944.502400:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016944.502404:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016944.502406:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016944.502411:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1207 00000100:00080000:3.0:1734016944.502414:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016944.502418:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016944.502428:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016944.502440:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332710400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016944.502451:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016944.502454:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016944.502456:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016944.502460:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1207 00000100:00080000:3.0:1734016944.502463:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016944.502466:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016944.502471:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016944.502479:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332710528/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016944.502488:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016944.502491:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016944.502493:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016944.502496:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1207 00000100:00080000:3.0:1734016944.502499:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016944.502502:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734016944.502506:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332710272:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734016944.502506:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016944.502514:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332710656/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016944.502523:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016944.502526:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016944.502527:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016944.502530:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1207 00000100:00080000:3.0:1734016944.502533:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016944.502536:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016944.502540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016944.502548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332710784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016944.502556:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016944.502559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016944.502560:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016944.502564:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1207 00000100:00080000:3.0:1734016944.502567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016944.502570:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734016944.502571:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734016944.502575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016944.502583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332710912/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016944.502591:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016944/real 1734016944] req@ffff8800a73cf480 x1818247332710272/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016944.502591:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016944.502594:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016944.502596:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016944.502599:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1207 00000100:00080000:3.0:1734016944.502602:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016944.502605:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016944.502609:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016944.502610:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332710272/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016944.502617:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332711040/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016944.502620:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016944.502623:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016944.502628:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332710272:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016944.502636:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332710400:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016944.502656:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016944.502663:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016944/real 1734016944] req@ffff8800a73cf800 x1818247332710400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016944.502674:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332710400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016944.502682:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016944.502684:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016944.502688:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332710400:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016944.502694:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332710528:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016944.502707:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016944.502712:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016944/real 1734016944] req@ffff8800a73cd500 x1818247332710528/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016944.502722:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332710528/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016944.502730:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016944.502732:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016944.502735:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332710528:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016944.502759:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332710656:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016944.502771:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016944.502777:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016944/real 1734016944] req@ffff8800a73cfb80 x1818247332710656/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016944.502794:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332710656/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016944.502802:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016944.502804:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016944.502807:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332710656:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016944.502812:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332710784:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016944.502823:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016944.502829:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016944/real 1734016944] req@ffff8800a73cd880 x1818247332710784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016944.502839:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332710784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016944.502846:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016944.502848:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016944.502851:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332710784:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016944.502856:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332710912:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016944.502868:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016944.502873:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016944/real 1734016944] req@ffff8800a73cca80 x1818247332710912/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016944.502883:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332710912/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016944.502890:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016944.502893:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016944.502896:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332710912:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016944.502902:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332711040:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016944.502914:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016944.502919:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016944/real 1734016944] req@ffff8800a73ced80 x1818247332711040/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016944.502929:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332711040/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016944.502936:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016944.502938:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016944.502941:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332711040:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016944.596387:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016945.598480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016946.600476:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016947.602484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016948.604512:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016949.510366:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016949.510379:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016949.510385:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016949.510399:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1212 00000100:00080000:3.0:1734016949.510409:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016949.510419:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016949.510469:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332711168/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016949.510505:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016949.510513:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016949.510515:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016949.510520:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1212 00000100:00080000:3.0:1734016949.510523:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016949.510528:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016949.510540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016949.510551:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332711296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016949.510562:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016949.510565:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016949.510567:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016949.510571:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1212 00000100:00080000:3.0:1734016949.510574:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016949.510578:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016949.510583:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016949.510592:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332711424/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016949.510601:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016949.510604:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016949.510606:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734016949.510610:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332711168:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016949.510610:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1212 00000100:00080000:3.0:1734016949.510613:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016949.510616:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016949.510621:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016949.510630:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332711552/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016949.510639:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016949.510642:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016949.510644:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016949.510647:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1212 00000100:00080000:3.0:1734016949.510650:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016949.510654:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016949.510658:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016949.510666:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332711680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016949.510675:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016949.510678:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016949.510679:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016949.510683:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1212 00000100:00080000:3.0:1734016949.510686:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016949.510690:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016949.510694:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016949.510702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332711808/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016949.510710:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016949.510713:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016949.510715:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016949.510719:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1212 00000100:00080000:3.0:1734016949.510721:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734016949.510723:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332711296:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734016949.510725:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016949.510730:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016949.510741:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332711936/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016949.510756:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332711424:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016949.510773:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332711552:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016949.510788:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332711680:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016949.510806:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332711808:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016949.510822:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332711936:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016949.606511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016950.608523:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016951.610527:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1734016952.516513:0:1955:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:1.0:1734016952.516561:0:1955:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1734016952.516581:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734016952.516616:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734016952.516625:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734016952.516631:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734016952.516639:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734016952.516645:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734016952.516651:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:0.0:1734016952.516725:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016949/real 1734016952] req@ffff8800a73cea00 x1818247332711424/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016952.516764:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332711424/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016952.516782:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016952.516787:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016952.516807:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332711424:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016952.516820:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016949/real 1734016952] req@ffff8800a73cdc00 x1818247332711552/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016952.516892:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332711552/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016952.516906:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016952.516911:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016952.516917:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332711552:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016952.516927:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016949/real 1734016952] req@ffff8800a73cf100 x1818247332711680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016952.516946:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332711680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016952.516959:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016952.516963:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016952.516968:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332711680:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016952.516979:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016949/real 1734016952] req@ffff8800a73cce00 x1818247332711808/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016952.517006:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332711808/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016952.517017:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016952.517021:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016952.517027:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332711808:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016952.517036:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016949/real 1734016952] req@ffff8800a73cc380 x1818247332711936/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016952.517052:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332711936/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016952.517064:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016952.517068:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016952.517073:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332711936:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016952.517083:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016949/real 1734016952] req@ffff8800a73cdf80 x1818247332711168/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016952.517103:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332711168/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016952.517114:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016952.517118:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016952.517123:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332711168:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016952.517133:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016949/real 1734016952] req@ffff8800a73cc700 x1818247332711296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016952.517162:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332711296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016952.517174:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016952.517178:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016952.517183:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332711296:192.168.201.152@tcp:38:kworker.0 00010000:00010000:3.0:1734016952.612485:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016953.614498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016954.518360:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016954.518370:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016954.518375:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016954.518385:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1217 00000100:00080000:3.0:1734016954.518394:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016954.518402:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016954.518439:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332712064/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016954.518472:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016954.518476:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016954.518478:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016954.518483:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1217 00000100:00080000:3.0:1734016954.518486:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016954.518490:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016954.518500:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016954.518513:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332712192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016954.518524:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016954.518527:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016954.518529:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016954.518533:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1217 00000100:00080000:3.0:1734016954.518535:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016954.518538:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016954.518543:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016954.518551:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332712320/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016954.518560:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016954.518563:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016954.518564:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016954.518568:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1217 00000100:00080000:3.0:1734016954.518570:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016954.518573:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016954.518578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016954.518583:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332712064:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734016954.518585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332712448/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016954.518594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016954.518597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016954.518599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016954.518602:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1217 00000100:00080000:3.0:1734016954.518605:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016954.518609:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016954.518613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016954.518620:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332712576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016954.518628:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016954.518631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016954.518632:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016954.518636:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1217 00000100:00080000:3.0:1734016954.518641:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016954.518644:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016954.518648:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734016954.518651:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734016954.518656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332712704/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016954.518663:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016954.518667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016954.518668:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016954.518671:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1217 00000100:00100000:0.0:1734016954.518672:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016954/real 1734016954] req@ffff8800a73ce680 x1818247332712064/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016954.518674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016954.518677:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016954.518681:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016954.518689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332712832/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016954.518692:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332712064/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016954.518702:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016954.518706:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016954.518711:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332712064:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016954.518718:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332712192:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016954.518736:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016954.518742:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016954/real 1734016954] req@ffff8800a73cc700 x1818247332712192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016954.518753:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332712192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016954.518761:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016954.518764:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016954.518768:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332712192:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016954.518775:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332712320:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016954.518832:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016954.518841:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016954/real 1734016954] req@ffff8800a73cdf80 x1818247332712320/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016954.518852:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332712320/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016954.518860:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016954.518863:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016954.518866:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332712320:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016954.518891:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332712448:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016954.518908:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016954.518914:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016954/real 1734016954] req@ffff8800a73cc380 x1818247332712448/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016954.518924:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332712448/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016954.518932:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016954.518934:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016954.518937:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332712448:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016954.518942:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332712576:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016954.518958:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016954.518964:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016954/real 1734016954] req@ffff8800a73cce00 x1818247332712576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016954.518974:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332712576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016954.518982:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016954.518984:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016954.518987:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332712576:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016954.518992:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332712704:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016954.519003:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016954.519009:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016954/real 1734016954] req@ffff8800a73cf100 x1818247332712704/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016954.519019:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332712704/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016954.519026:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016954.519028:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016954.519031:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332712704:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016954.519036:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332712832:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016954.519048:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016954.519054:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016954/real 1734016954] req@ffff8800a73cdc00 x1818247332712832/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016954.519071:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332712832/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016954.519079:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016954.519081:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016954.519084:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332712832:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016954.616466:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016955.618503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016956.620516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016957.622436:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016958.624522:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016959.526363:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016959.526373:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016959.526377:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016959.526388:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1222 00000100:00080000:3.0:1734016959.526399:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016959.526407:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016959.526442:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332712960/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016959.526477:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016959.526481:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016959.526483:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016959.526488:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1222 00000100:00080000:3.0:1734016959.526491:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016959.526495:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016959.526504:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016959.526516:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332713088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016959.526526:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016959.526529:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016959.526531:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016959.526534:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1222 00000100:00080000:3.0:1734016959.526538:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016959.526541:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016959.526546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016959.526555:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332713216/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016959.526563:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016959.526566:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016959.526568:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016959.526571:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1222 00000100:00080000:3.0:1734016959.526574:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016959.526577:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016959.526581:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016959.526588:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332713344/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016959.526598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016959.526601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016959.526603:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016959.526607:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1222 00000100:00100000:0.0:1734016959.526608:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332712960:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016959.526609:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016959.526613:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016959.526617:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016959.526625:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332713472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016959.526640:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016959.526643:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016959.526645:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016959.526649:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1222 00000100:00080000:3.0:1734016959.526651:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016959.526655:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016959.526659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016959.526667:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332713600/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016959.526675:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016959.526678:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016959.526679:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016959.526682:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1222 00000100:00080000:3.0:1734016959.526686:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016959.526689:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016959.526693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734016959.526698:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734016959.526702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332713728/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016959.526763:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016959/real 1734016959] req@ffff8800a73cea00 x1818247332712960/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016959.526787:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332712960/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016959.526800:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016959.526804:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016959.526809:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332712960:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016959.526818:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332713088:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016959.526840:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016959.526847:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016959/real 1734016959] req@ffff8800a73ced80 x1818247332713088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016959.526858:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332713088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016959.526867:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016959.526871:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016959.526875:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332713088:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016959.526882:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332713216:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016959.526897:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016959.526904:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016959/real 1734016959] req@ffff8800a73cca80 x1818247332713216/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016959.526914:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332713216/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016959.526922:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016959.526925:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016959.526928:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332713216:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016959.526960:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332713344:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016959.526973:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016959.526980:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016959/real 1734016959] req@ffff8800a73cd880 x1818247332713344/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016959.526990:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332713344/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016959.526999:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016959.527001:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016959.527004:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332713344:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016959.527010:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332713472:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016959.527026:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016959.527033:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016959/real 1734016959] req@ffff8800a73cfb80 x1818247332713472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016959.527043:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332713472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016959.527052:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016959.527054:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016959.527057:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332713472:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016959.527063:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332713600:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016959.527076:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016959.527082:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016959/real 1734016959] req@ffff8800a73cd500 x1818247332713600/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016959.527092:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332713600/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016959.527099:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016959.527101:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016959.527104:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332713600:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016959.527110:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332713728:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016959.527124:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016959.527130:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016959/real 1734016959] req@ffff8800a73cf800 x1818247332713728/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016959.527147:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332713728/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016959.527156:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016959.527158:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016959.527161:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332713728:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016959.626499:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016960.628487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016961.630500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016962.632491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016963.634464:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016964.534366:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016964.534376:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016964.534381:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016964.534391:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1227 00000100:00080000:3.0:1734016964.534403:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016964.534410:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016964.534448:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332713856/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016964.534484:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016964.534489:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016964.534491:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016964.534495:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1227 00000100:00080000:3.0:1734016964.534499:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016964.534503:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016964.534512:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016964.534525:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332713984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016964.534536:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016964.534539:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016964.534541:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016964.534545:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1227 00000100:00080000:3.0:1734016964.534547:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016964.534551:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016964.534556:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016964.534564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332714112/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016964.534573:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016964.534577:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016964.534579:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016964.534582:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1227 00000100:00080000:3.0:1734016964.534585:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016964.534588:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016964.534592:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016964.534600:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332714240/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016964.534609:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016964.534613:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1734016964.534614:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332713856:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016964.534614:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016964.534653:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1227 00000100:00080000:3.0:1734016964.534656:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016964.534660:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016964.534665:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016964.534675:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332714368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016964.534684:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016964.534687:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016964.534688:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016964.534692:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1227 00000100:00080000:3.0:1734016964.534695:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016964.534699:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016964.534703:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016964.534711:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332714496/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016964.534719:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016964.534723:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016964.534724:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016964.534728:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1227 00000100:00080000:3.0:1734016964.534731:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016964.534734:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016964.534738:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016964.534746:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332714624/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734016964.534751:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016964.534780:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016964/real 1734016964] req@ffff8800a73cf480 x1818247332713856/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016964.534804:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332713856/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016964.534817:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016964.534821:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016964.534826:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332713856:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016964.534834:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332713984:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016964.534854:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016964.534861:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016964/real 1734016964] req@ffff8800a73cf800 x1818247332713984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016964.534873:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332713984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016964.534881:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016964.534886:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016964.534889:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332713984:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016964.534896:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332714112:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016964.534912:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016964.534918:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016964/real 1734016964] req@ffff8800a73cd500 x1818247332714112/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016964.534928:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332714112/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016964.534936:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016964.534939:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016964.534942:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332714112:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016964.534971:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332714240:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016964.534984:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016964.534990:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016964/real 1734016964] req@ffff8800a73cfb80 x1818247332714240/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016964.535000:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332714240/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016964.535009:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016964.535011:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016964.535014:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332714240:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016964.535020:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332714368:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016964.535037:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016964.535043:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016964/real 1734016964] req@ffff8800a73cd880 x1818247332714368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016964.535054:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332714368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016964.535061:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016964.535064:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016964.535067:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332714368:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016964.535073:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332714496:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016964.535086:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016964.535092:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016964/real 1734016964] req@ffff8800a73cca80 x1818247332714496/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016964.535101:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332714496/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016964.535108:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016964.535111:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016964.535114:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332714496:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016964.535120:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332714624:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016964.535134:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016964.535140:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016964/real 1734016964] req@ffff8800a73ced80 x1818247332714624/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016964.535157:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332714624/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016964.535165:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016964.535167:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016964.535170:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332714624:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016964.636487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016965.638397:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016966.640492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016967.642509:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016968.644492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016969.542292:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016969.542302:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016969.542307:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016969.542318:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1232 00000100:00080000:3.0:1734016969.542329:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016969.542336:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016969.542372:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332714752/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016969.542405:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016969.542412:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016969.542414:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016969.542418:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1232 00000100:00080000:3.0:1734016969.542422:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016969.542426:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016969.542436:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016969.542447:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332714880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016969.542457:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016969.542461:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016969.542462:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016969.542466:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1232 00000100:00080000:3.0:1734016969.542468:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016969.542472:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016969.542476:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016969.542484:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332715008/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016969.542492:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016969.542496:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016969.542498:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016969.542501:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1232 00000100:00080000:3.0:1734016969.542503:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016969.542507:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016969.542512:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016969.542520:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332715136/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016969.542530:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016969.542575:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016969.542577:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016969.542581:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1232 00000100:00080000:3.0:1734016969.542584:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734016969.542586:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332714752:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016969.542588:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016969.542593:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016969.542603:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332715264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016969.542613:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016969.542616:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016969.542617:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016969.542621:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1232 00000100:00080000:3.0:1734016969.542624:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016969.542627:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016969.542632:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016969.542639:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332715392/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016969.542647:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016969.542651:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016969.542653:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016969.542656:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1232 00000100:00080000:3.0:1734016969.542660:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016969.542663:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016969.542667:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016969.542676:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332715520/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734016969.542680:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016969.542709:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016969/real 1734016969] req@ffff8800a73cea00 x1818247332714752/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016969.542731:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332714752/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016969.542744:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016969.542748:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016969.542753:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332714752:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016969.542762:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332714880:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016969.542782:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016969.542789:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016969/real 1734016969] req@ffff8800a73cdc00 x1818247332714880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016969.542801:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332714880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016969.542809:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016969.542814:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016969.542817:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332714880:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016969.542824:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332715008:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016969.542840:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016969.542847:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016969/real 1734016969] req@ffff8800a73cf100 x1818247332715008/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016969.542857:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332715008/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016969.542865:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016969.542868:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016969.542871:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332715008:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016969.542902:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332715136:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016969.542915:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016969.542921:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016969/real 1734016969] req@ffff8800a73cce00 x1818247332715136/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016969.542932:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332715136/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016969.542941:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016969.542943:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016969.542946:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332715136:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016969.542952:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332715264:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016969.542969:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016969.542975:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016969/real 1734016969] req@ffff8800a73cc380 x1818247332715264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016969.542986:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332715264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016969.542993:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016969.542996:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016969.542999:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332715264:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016969.543006:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332715392:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016969.543019:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016969.543024:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016969/real 1734016969] req@ffff8800a73cdf80 x1818247332715392/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016969.543034:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332715392/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016969.543043:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016969.543045:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016969.543048:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332715392:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016969.543053:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332715520:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016969.543067:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016969.543073:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016969/real 1734016969] req@ffff8800a73cc700 x1818247332715520/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016969.543090:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332715520/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016969.543098:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016969.543100:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016969.543103:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332715520:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016969.646501:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016970.648562:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016971.650546:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016972.652526:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016973.654532:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016974.550343:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016974.550353:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016974.550358:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016974.550368:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1237 00000100:00080000:3.0:1734016974.550379:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016974.550387:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016974.550425:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332715648/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016974.550494:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016974.550499:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016974.550501:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016974.550506:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1237 00000100:00080000:3.0:1734016974.550510:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016974.550514:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016974.550524:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016974.550539:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332715776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016974.550551:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016974.550554:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016974.550556:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016974.550559:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1237 00000100:00080000:3.0:1734016974.550562:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016974.550565:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016974.550570:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016974.550578:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332715904/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016974.550588:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016974.550592:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016974.550593:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734016974.550594:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332715648:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016974.550597:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1237 00000100:00080000:3.0:1734016974.550599:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016974.550603:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016974.550608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016974.550616:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332716032/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016974.550624:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016974.550628:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016974.550629:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016974.550633:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1237 00000100:00080000:3.0:1734016974.550636:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016974.550640:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016974.550644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016974.550652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332716160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016974.550660:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016974.550663:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016974.550665:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016974.550669:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1237 00000100:00080000:3.0:1734016974.550672:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016974.550675:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016974.550679:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734016974.550683:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734016974.550686:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332716288/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016974.550694:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016974.550698:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016974.550699:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016974.550702:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1237 00000100:00080000:3.0:1734016974.550705:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016974.550708:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734016974.550712:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016974/real 1734016974] req@ffff8800a73ce680 x1818247332715648/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734016974.550712:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016974.550721:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332716416/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016974.550735:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332715648/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734016999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016974.550748:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016974.550752:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016974.550757:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332715648:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016974.550766:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332715776:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016974.550787:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016974.550794:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016974/real 1734016974] req@ffff8800a73cc700 x1818247332715776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016974.550806:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332715776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016974.550814:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016974.550818:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016974.550822:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332715776:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016974.550850:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332715904:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016974.550866:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016974.550872:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016974/real 1734016974] req@ffff8800a73cdf80 x1818247332715904/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016974.550884:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332715904/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016974.550891:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016974.550894:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016974.550897:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332715904:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016974.550903:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332716032:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016974.550915:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016974.550921:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016974/real 1734016974] req@ffff8800a73cc380 x1818247332716032/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016974.550931:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332716032/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016974.550938:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016974.550941:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016974.550944:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332716032:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016974.550951:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332716160:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016974.550968:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016974.550974:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016974/real 1734016974] req@ffff8800a73cce00 x1818247332716160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016974.550985:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332716160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734016999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016974.550992:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016974.550995:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016974.550998:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332716160:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016974.551003:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332716288:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016974.551016:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016974.551022:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016974/real 1734016974] req@ffff8800a73cf100 x1818247332716288/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016974.551032:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332716288/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016974.551040:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016974.551043:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016974.551046:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332716288:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016974.551051:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332716416:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016974.551065:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016974.551070:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016974/real 1734016974] req@ffff8800a73cdc00 x1818247332716416/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016974.551087:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332716416/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734016999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016974.551095:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016974.551098:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016974.551101:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332716416:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016974.656494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016975.658536:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016976.660462:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016977.662500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016978.664484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016979.558412:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016979.558421:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016979.558425:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016979.558437:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1242 00000100:00080000:3.0:1734016979.558447:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016979.558455:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016979.558492:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332716544/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016979.558527:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016979.558532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016979.558534:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016979.558538:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1242 00000100:00080000:3.0:1734016979.558541:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016979.558545:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016979.558555:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016979.558566:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332716672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016979.558576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016979.558580:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016979.558582:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016979.558585:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1242 00000100:00080000:3.0:1734016979.558588:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016979.558592:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016979.558597:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016979.558606:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332716800/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016979.558615:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016979.558618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016979.558619:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016979.558622:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1242 00000100:00080000:3.0:1734016979.558625:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016979.558628:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016979.558632:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016979.558641:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332716928/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016979.558650:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332716544:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734016979.558650:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016979.558654:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016979.558655:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016979.558659:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1242 00000100:00080000:3.0:1734016979.558661:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016979.558665:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016979.558669:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016979.558678:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332717056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016979.558692:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016979.558695:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016979.558697:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016979.558701:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1242 00000100:00080000:3.0:1734016979.558704:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016979.558707:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016979.558711:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016979.558718:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332717184/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016979.558726:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016979.558729:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016979.558730:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016979.558733:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1242 00000100:00080000:3.0:1734016979.558737:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734016979.558739:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016979.558740:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016979.558744:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016979.558753:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332717312/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016979.558767:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016979/real 1734016979] req@ffff8800a73cea00 x1818247332716544/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016979.558790:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332716544/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016979.558804:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016979.558808:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016979.558813:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332716544:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016979.558822:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332716672:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016979.558841:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016979.558848:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016979/real 1734016979] req@ffff8800a73ced80 x1818247332716672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016979.558859:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332716672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016979.558867:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016979.558871:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016979.558875:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332716672:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016979.558882:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332716800:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016979.558897:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016979.558903:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016979/real 1734016979] req@ffff8800a73cca80 x1818247332716800/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016979.558913:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332716800/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016979.558921:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016979.558924:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016979.558928:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332716800:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016979.558958:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332716928:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016979.558971:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016979.558977:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016979/real 1734016979] req@ffff8800a73cd880 x1818247332716928/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016979.558988:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332716928/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016979.558996:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016979.558998:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016979.559001:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332716928:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016979.559007:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332717056:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016979.559024:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016979.559030:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016979/real 1734016979] req@ffff8800a73cfb80 x1818247332717056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016979.559040:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332717056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016979.559048:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016979.559050:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016979.559053:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332717056:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016979.559059:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332717184:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016979.559071:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016979.559078:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016979/real 1734016979] req@ffff8800a73cd500 x1818247332717184/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016979.559088:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332717184/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016979.559096:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016979.559099:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016979.559102:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332717184:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016979.559108:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332717312:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016979.559121:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016979.559127:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016979/real 1734016979] req@ffff8800a73cf800 x1818247332717312/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016979.559145:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332717312/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016979.559153:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016979.559156:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016979.559159:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332717312:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016979.666539:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016980.668531:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016981.670533:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016982.672516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016983.674533:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016984.566385:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016984.566395:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016984.566400:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016984.566410:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1247 00000100:00080000:3.0:1734016984.566420:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016984.566428:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016984.566464:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332717440/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016984.566496:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016984.566500:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016984.566502:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016984.566507:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1247 00000100:00080000:3.0:1734016984.566510:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016984.566514:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016984.566524:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016984.566536:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332717568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016984.566546:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016984.566549:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016984.566551:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016984.566555:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1247 00000100:00080000:3.0:1734016984.566558:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016984.566561:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016984.566567:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016984.566575:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332717696/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016984.566583:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016984.566587:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016984.566588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016984.566592:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1247 00000100:00080000:3.0:1734016984.566594:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016984.566598:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016984.566602:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016984.566606:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332717440:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734016984.566610:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332717824/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016984.566619:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016984.566622:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016984.566623:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016984.566627:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1247 00000100:00080000:3.0:1734016984.566629:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016984.566632:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016984.566637:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016984.566644:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332717952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016984.566652:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016984.566655:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016984.566657:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016984.566661:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1247 00000100:00080000:3.0:1734016984.566664:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016984.566667:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734016984.566671:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734016984.566672:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016984.566679:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332718080/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016984.566688:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016984.566691:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734016984.566692:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016984/real 1734016984] req@ffff8800a73cf480 x1818247332717440/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016984.566692:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016984.566696:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1247 00000100:00080000:3.0:1734016984.566699:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016984.566702:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016984.566706:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016984.566712:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332717440/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016984.566714:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332718208/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016984.566723:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016984.566726:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016984.566731:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332717440:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016984.566738:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332717568:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016984.566756:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016984.566763:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016984/real 1734016984] req@ffff8800a73cf800 x1818247332717568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016984.566774:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332717568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016984.566782:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016984.566785:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016984.566789:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332717568:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016984.566795:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332717696:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016984.566809:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016984.566815:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016984/real 1734016984] req@ffff8800a73cd500 x1818247332717696/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016984.566825:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332717696/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016984.566833:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016984.566835:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016984.566839:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332717696:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016984.566863:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332717824:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016984.566875:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016984.566881:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016984/real 1734016984] req@ffff8800a73cfb80 x1818247332717824/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016984.566892:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332717824/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016984.566899:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016984.566901:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016984.566904:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332717824:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016984.566909:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332717952:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016984.566924:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016984.566930:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016984/real 1734016984] req@ffff8800a73cd880 x1818247332717952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016984.566940:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332717952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016984.566948:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016984.566951:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016984.566954:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332717952:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016984.566959:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332718080:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016984.566970:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016984.566976:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016984/real 1734016984] req@ffff8800a73cca80 x1818247332718080/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016984.566985:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332718080/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016984.566992:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016984.566994:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016984.566998:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332718080:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016984.567003:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332718208:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016984.567015:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016984.567021:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016984/real 1734016984] req@ffff8800a73ced80 x1818247332718208/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016984.567037:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332718208/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016984.567044:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016984.567046:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016984.567049:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332718208:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016984.676500:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016985.678472:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016986.680491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016987.682502:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016988.684448:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016989.574341:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016989.574351:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016989.574356:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016989.574366:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1252 00000100:00080000:3.0:1734016989.574376:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016989.574384:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016989.574420:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332718336/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016989.574453:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016989.574459:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016989.574462:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016989.574466:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1252 00000100:00080000:3.0:1734016989.574469:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016989.574473:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016989.574483:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016989.574494:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332718464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016989.574504:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016989.574507:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016989.574509:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016989.574512:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1252 00000100:00080000:3.0:1734016989.574515:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016989.574518:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016989.574523:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016989.574531:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332718592/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016989.574539:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016989.574542:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016989.574544:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016989.574547:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1252 00000100:00080000:3.0:1734016989.574550:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016989.574553:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016989.574558:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016989.574561:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332718336:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734016989.574566:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332718720/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016989.574575:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016989.574579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016989.574580:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016989.574583:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1252 00000100:00080000:3.0:1734016989.574586:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016989.574589:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016989.574593:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016989.574601:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332718848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016989.574609:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016989.574612:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016989.574613:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016989.574617:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1252 00000100:00080000:3.0:1734016989.574620:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016989.574623:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734016989.574627:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734016989.574627:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016989.574635:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332718976/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016989.574643:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016989.574646:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016989.574648:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734016989.574649:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016989/real 1734016989] req@ffff8800a73cea00 x1818247332718336/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016989.574651:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1252 00000100:00080000:3.0:1734016989.574654:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016989.574657:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016989.574661:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016989.574668:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332718336/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016989.574671:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332719104/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016989.574679:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016989.574682:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016989.574687:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332718336:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016989.574694:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332718464:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016989.574711:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016989.574718:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016989/real 1734016989] req@ffff8800a73cdc00 x1818247332718464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016989.574729:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332718464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016989.574736:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016989.574740:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016989.574744:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332718464:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016989.574750:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332718592:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016989.574763:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016989.574769:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016989/real 1734016989] req@ffff8800a73cf100 x1818247332718592/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016989.574779:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332718592/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016989.574787:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016989.574789:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016989.574793:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332718592:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016989.574817:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332718720:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016989.574829:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016989.574835:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016989/real 1734016989] req@ffff8800a73cce00 x1818247332718720/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016989.574846:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332718720/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016989.574853:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016989.574855:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016989.574858:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332718720:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016989.574863:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332718848:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016989.574878:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016989.574884:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016989/real 1734016989] req@ffff8800a73cc380 x1818247332718848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016989.574895:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332718848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016989.574902:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016989.574904:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016989.574907:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332718848:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016989.574912:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332718976:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016989.574924:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016989.574929:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016989/real 1734016989] req@ffff8800a73cdf80 x1818247332718976/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016989.574939:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332718976/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016989.574946:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016989.574948:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016989.574951:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332718976:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016989.574956:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332719104:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016989.574968:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016989.574974:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016989/real 1734016989] req@ffff8800a73cc700 x1818247332719104/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016989.574990:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332719104/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016989.574998:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016989.575000:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016989.575003:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332719104:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016989.686490:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016990.688497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016991.690495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016992.692466:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016993.694462:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016994.582367:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016994.582377:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016994.582381:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016994.582392:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1257 00000100:00080000:3.0:1734016994.582402:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016994.582410:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016994.582447:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332719232/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016994.582481:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016994.582485:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016994.582488:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016994.582492:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1257 00000100:00080000:3.0:1734016994.582495:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016994.582499:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016994.582509:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016994.582522:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332719360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016994.582532:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016994.582536:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016994.582537:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016994.582541:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1257 00000100:00080000:3.0:1734016994.582544:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016994.582547:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016994.582552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016994.582560:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332719488/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016994.582568:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016994.582571:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016994.582573:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016994.582576:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1257 00000100:00080000:3.0:1734016994.582578:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016994.582582:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016994.582586:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016994.582589:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332719232:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734016994.582594:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332719616/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016994.582603:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016994.582606:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016994.582608:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016994.582611:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1257 00000100:00080000:3.0:1734016994.582614:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016994.582617:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016994.582622:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016994.582629:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332719744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016994.582637:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016994.582640:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016994.582642:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016994.582645:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1257 00000100:00080000:3.0:1734016994.582648:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016994.582651:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734016994.582655:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734016994.582655:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016994.582663:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332719872/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016994.582671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016994.582674:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016994.582675:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734016994.582677:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016994/real 1734016994] req@ffff8800a73ce680 x1818247332719232/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016994.582679:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1257 00000100:00080000:3.0:1734016994.582681:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016994.582685:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016994.582689:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016994.582696:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332719232/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016994.582697:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332720000/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016994.582706:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016994.582710:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016994.582714:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332719232:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016994.582722:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332719360:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016994.582739:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016994.582746:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016994/real 1734016994] req@ffff8800a73cc700 x1818247332719360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016994.582757:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332719360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016994.582765:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016994.582768:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016994.582772:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332719360:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016994.582778:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332719488:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016994.582792:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016994.582798:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016994/real 1734016994] req@ffff8800a73cdf80 x1818247332719488/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016994.582808:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332719488/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016994.582816:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016994.582818:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016994.582822:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332719488:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016994.582846:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332719616:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016994.582859:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016994.582865:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016994/real 1734016994] req@ffff8800a73cc380 x1818247332719616/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016994.582875:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332719616/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016994.582883:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016994.582886:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016994.582888:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332719616:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016994.582894:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332719744:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016994.582909:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016994.582916:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016994/real 1734016994] req@ffff8800a73cce00 x1818247332719744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016994.582926:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332719744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016994.582933:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016994.582935:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016994.582938:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332719744:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016994.582943:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332719872:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016994.582954:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016994.582960:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016994/real 1734016994] req@ffff8800a73cf100 x1818247332719872/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016994.582970:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332719872/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016994.582978:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016994.582980:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016994.582983:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332719872:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016994.582988:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332720000:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016994.583000:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016994.583006:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016994/real 1734016994] req@ffff8800a73cdc00 x1818247332720000/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016994.583022:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332720000/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016994.583030:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016994.583032:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016994.583035:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332720000:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734016994.696476:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016995.698506:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016996.700504:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734016997.702477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734016998.704507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734016999.590356:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016999.590366:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734016999.590370:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016999.590380:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1262 00000100:00080000:3.0:1734016999.590389:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016999.590398:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734016999.590431:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332720128/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016999.590464:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016999.590468:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016999.590470:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016999.590474:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1262 00000100:00080000:3.0:1734016999.590478:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016999.590482:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016999.590491:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016999.590502:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332720256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016999.590512:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016999.590515:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016999.590517:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016999.590521:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1262 00000100:00080000:3.0:1734016999.590524:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016999.590528:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016999.590532:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734016999.590541:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332720384/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016999.590549:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016999.590552:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016999.590554:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016999.590557:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1262 00000100:00080000:3.0:1734016999.590560:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016999.590563:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016999.590567:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016999.590573:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332720128:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734016999.590575:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332720512/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016999.590584:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016999.590587:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734016999.590589:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016999.590592:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1262 00000100:00080000:3.0:1734016999.590595:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016999.590598:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016999.590602:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734016999.590610:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332720640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734016999.590625:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016999.590628:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734016999.590630:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016999.590633:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1262 00000100:00080000:3.0:1734016999.590636:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734016999.590638:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734016999.590640:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016999.590644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734016999.590651:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332720768/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734016999.590658:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016999/real 1734016999] req@ffff8800a73cea00 x1818247332720128/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734016999.590659:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734016999.590662:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734016999.590664:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734016999.590667:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1262 00000100:00080000:3.0:1734016999.590670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734016999.590673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734016999.590677:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734016999.590678:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332720128/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734016999.590686:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332720896/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734016999.590689:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016999.590693:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016999.590698:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332720128:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734016999.590705:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332720256:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016999.590722:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016999.590729:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016999/real 1734016999] req@ffff8800a73ced80 x1818247332720256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016999.590740:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332720256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016999.590748:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016999.590751:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016999.590755:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332720256:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016999.590761:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332720384:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016999.590774:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016999.590780:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016999/real 1734016999] req@ffff8800a73cca80 x1818247332720384/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016999.590790:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332720384/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016999.590798:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016999.590801:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016999.590804:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332720384:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016999.590829:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332720512:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016999.590841:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016999.590847:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016999/real 1734016999] req@ffff8800a73cd880 x1818247332720512/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016999.590858:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332720512/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016999.590866:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016999.590868:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016999.590871:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332720512:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016999.590876:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332720640:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734016999.590891:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016999.590897:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016999/real 1734016999] req@ffff8800a73cfb80 x1818247332720640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016999.590908:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332720640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016999.590916:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016999.590918:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016999.590921:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332720640:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734016999.590925:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332720768:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016999.590937:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016999.590943:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016999/real 1734016999] req@ffff8800a73cd500 x1818247332720768/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016999.590952:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332720768/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016999.590959:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016999.590961:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016999.590964:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332720768:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734016999.590970:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332720896:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734016999.590982:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734016999.590988:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734016999/real 1734016999] req@ffff8800a73cf800 x1818247332720896/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734016999.591005:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332720896/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734016999.591012:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734016999.591014:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734016999.591017:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332720896:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734016999.706458:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017000.708493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017001.710434:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017002.712497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017003.714458:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017004.598331:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017004.598341:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017004.598346:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017004.598356:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1267 00000100:00080000:3.0:1734017004.598365:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017004.598373:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017004.598410:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332721024/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017004.598444:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017004.598448:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017004.598450:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017004.598455:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1267 00000100:00080000:3.0:1734017004.598458:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017004.598462:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017004.598471:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017004.598483:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332721152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017004.598493:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017004.598497:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017004.598498:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017004.598502:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1267 00000100:00080000:3.0:1734017004.598505:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017004.598508:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017004.598513:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017004.598522:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332721280/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017004.598531:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017004.598534:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017004.598536:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017004.598539:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332721024:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017004.598540:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1267 00000100:00080000:3.0:1734017004.598543:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017004.598546:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017004.598550:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017004.598558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332721408/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017004.598565:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017004.598569:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017004.598570:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017004.598573:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1267 00000100:00080000:3.0:1734017004.598576:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017004.598579:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017004.598583:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017004.598590:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332721536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017004.598598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017004.598601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017004.598603:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017004.598607:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1267 00000100:00080000:3.0:1734017004.598610:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017004.598613:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017004.598618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017004.598619:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017004.598626:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332721664/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017004.598634:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017004.598637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017004.598639:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017004.598642:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1267 00000100:00080000:3.0:1734017004.598644:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017004.598646:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017004/real 1734017004] req@ffff8800a73cf480 x1818247332721024/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017004.598648:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017004.598652:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017004.598660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332721792/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017004.598673:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332721024/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017004.598689:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017004.598694:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017004.598701:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332721024:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017004.598712:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332721152:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017004.598743:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017004.598753:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017004/real 1734017004] req@ffff8800a73cf800 x1818247332721152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017004.598765:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332721152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017004.598774:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017004.598777:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017004.598781:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332721152:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017004.598805:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332721280:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017004.598820:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017004.598827:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017004/real 1734017004] req@ffff8800a73cd500 x1818247332721280/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017004.598838:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332721280/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017004.598845:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017004.598847:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017004.598850:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332721280:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017004.598855:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332721408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017004.598867:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017004.598872:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017004/real 1734017004] req@ffff8800a73cfb80 x1818247332721408/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017004.598882:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332721408/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017004.598889:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017004.598891:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017004.598895:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332721408:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017004.598900:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332721536:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017004.598916:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017004.598922:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017004/real 1734017004] req@ffff8800a73cd880 x1818247332721536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017004.598932:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332721536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017004.598974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017004.598976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017004.598980:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332721536:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017004.598986:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332721664:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017004.599002:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017004.599009:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017004/real 1734017004] req@ffff8800a73cca80 x1818247332721664/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017004.599019:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332721664/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017004.599027:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017004.599030:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017004.599033:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332721664:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017004.599038:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332721792:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017004.599050:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017004.599055:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017004/real 1734017004] req@ffff8800a73ced80 x1818247332721792/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017004.599072:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332721792/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017004.599080:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017004.599082:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017004.599085:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332721792:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017004.716449:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017005.718498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017006.720501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017007.722489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017008.724480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017009.606379:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017009.606389:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017009.606394:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017009.606404:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1272 00000100:00080000:3.0:1734017009.606418:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017009.606426:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017009.606463:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332721920/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017009.606497:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017009.606503:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017009.606505:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017009.606510:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1272 00000100:00080000:3.0:1734017009.606513:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017009.606517:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017009.606526:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017009.606537:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332722048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017009.606547:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017009.606551:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017009.606552:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017009.606556:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1272 00000100:00080000:3.0:1734017009.606558:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017009.606562:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017009.606567:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017009.606574:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332722176/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017009.606583:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017009.606586:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017009.606587:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017009.606591:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1272 00000100:00080000:3.0:1734017009.606593:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017009.606598:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017009.606602:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017009.606603:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332721920:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017009.606610:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332722304/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017009.606619:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017009.606622:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017009.606624:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017009.606627:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1272 00000100:00080000:3.0:1734017009.606630:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017009.606633:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017009.606637:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017009.606645:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332722432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017009.606653:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017009.606656:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017009.606657:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017009.606661:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1272 00000100:00080000:3.0:1734017009.606663:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734017009.606666:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017009.606667:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017009.606671:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017009.606678:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332722560/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017009.606686:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017009.606687:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017009/real 1734017009] req@ffff8800a73cea00 x1818247332721920/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017009.606690:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017009.606692:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017009.606696:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1272 00000100:00080000:3.0:1734017009.606698:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017009.606701:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017009.606706:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017009.606707:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332721920/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017009.606714:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332722688/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017009.606718:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017009.606721:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017009.606726:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332721920:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017009.606733:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332722048:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017009.606750:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017009.606757:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017009/real 1734017009] req@ffff8800a73cdc00 x1818247332722048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017009.606768:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332722048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017009.606776:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017009.606780:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017009.606783:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332722048:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017009.606789:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332722176:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017009.606802:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017009.606808:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017009/real 1734017009] req@ffff8800a73cf100 x1818247332722176/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017009.606818:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332722176/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017009.606826:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017009.606828:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017009.606831:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332722176:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017009.606903:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332722304:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017009.606921:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017009.606929:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017009/real 1734017009] req@ffff8800a73cce00 x1818247332722304/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017009.606940:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332722304/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017009.606948:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017009.606950:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017009.606954:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332722304:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017009.606959:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332722432:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017009.606974:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017009.606980:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017009/real 1734017009] req@ffff8800a73cc380 x1818247332722432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017009.606990:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332722432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017009.606998:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017009.607000:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017009.607003:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332722432:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017009.607008:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332722560:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017009.607019:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017009.607025:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017009/real 1734017009] req@ffff8800a73cdf80 x1818247332722560/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017009.607035:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332722560/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017009.607042:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017009.607044:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017009.607047:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332722560:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017009.607052:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332722688:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017009.607064:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017009.607070:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017009/real 1734017009] req@ffff8800a73cc700 x1818247332722688/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017009.607087:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332722688/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017009.607094:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017009.607096:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017009.607099:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332722688:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017009.726510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017010.728479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017011.730463:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017012.732479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017013.734508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017014.614368:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017014.614378:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017014.614384:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017014.614394:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1277 00000100:00080000:3.0:1734017014.614405:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017014.614413:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017014.614451:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332722816/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017014.614485:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017014.614490:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017014.614492:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017014.614496:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1277 00000100:00080000:3.0:1734017014.614500:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017014.614504:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017014.614514:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017014.614527:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332722944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017014.614539:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017014.614542:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017014.614544:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017014.614547:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1277 00000100:00080000:3.0:1734017014.614549:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017014.614553:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017014.614557:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017014.614566:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332723072/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017014.614575:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017014.614578:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017014.614579:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017014.614583:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1277 00000100:00080000:3.0:1734017014.614585:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017014.614589:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017014.614593:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017014.614601:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332723200/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017014.614610:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017014.614614:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017014.614615:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017014.614619:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1277 00000100:00080000:3.0:1734017014.614622:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017014.614625:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332722816:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017014.614625:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017014.614630:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017014.614638:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332723328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017014.614646:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017014.614649:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017014.614650:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017014.614654:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1277 00000100:00080000:3.0:1734017014.614657:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017014.614660:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017014.614665:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017014.614672:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332723456/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017014.614680:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017014.614683:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017014.614685:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017014.614688:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1277 00000100:00080000:3.0:1734017014.614691:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017014.614694:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017014.614698:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017014.614706:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332723584/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017014.614745:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332722944:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017014.614812:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332723072:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017014.614844:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332723200:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017014.614861:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332723328:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017014.614877:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332723456:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017014.614893:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332723584:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017014.736491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017015.738481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017016.740484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1734017017.620482:0:1957:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:1.0:1734017017.620526:0:1957:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1734017017.620545:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017017.620579:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017017.620586:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017017.620592:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017017.620599:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017017.620605:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017017.620611:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:0.0:1734017017.620676:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017014/real 1734017017] req@ffff8800a73cc380 x1818247332723200/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017017.620705:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332723200/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017017.620762:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017017.620767:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017017.620772:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332723200:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017017.620781:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017014/real 1734017017] req@ffff8800a73cce00 x1818247332723328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017017.620794:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332723328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017017.620802:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017017.620805:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017017.620811:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332723328:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017017.620822:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017014/real 1734017017] req@ffff8800a73cf100 x1818247332723456/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017017.620841:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332723456/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017017.620852:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017017.620856:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017017.620861:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332723456:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017017.620871:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017014/real 1734017017] req@ffff8800a73cdc00 x1818247332723584/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017017.620888:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332723584/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017017.620900:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017017.620904:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017017.620909:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332723584:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017017.620924:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017014/real 1734017017] req@ffff8800a73ce680 x1818247332722816/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017017.620943:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332722816/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017017.620955:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017017.620959:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017017.620964:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332722816:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017017.620974:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017014/real 1734017017] req@ffff8800a73cc700 x1818247332722944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017017.620992:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332722944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017017.621004:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017017.621008:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017017.621014:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332722944:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017017.621024:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017014/real 1734017017] req@ffff8800a73cdf80 x1818247332723072/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017017.621041:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332723072/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017017.621055:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017017.621059:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017017.621064:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332723072:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017017.742485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017018.744463:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017019.622349:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017019.622359:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017019.622364:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017019.622374:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1282 00000100:00080000:3.0:1734017019.622385:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017019.622392:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017019.622428:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332723712/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017019.622462:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017019.622466:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017019.622468:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017019.622472:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1282 00000100:00080000:3.0:1734017019.622475:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017019.622479:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017019.622489:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017019.622499:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332723840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017019.622509:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017019.622513:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017019.622514:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017019.622518:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1282 00000100:00080000:3.0:1734017019.622521:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017019.622525:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017019.622529:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017019.622537:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332723968/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017019.622546:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017019.622549:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017019.622550:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017019.622553:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1282 00000100:00080000:3.0:1734017019.622556:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017019.622559:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017019.622564:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017019.622567:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332723712:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017019.622571:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332724096/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017019.622580:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017019.622583:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017019.622585:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017019.622588:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1282 00000100:00080000:3.0:1734017019.622590:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017019.622594:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017019.622598:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017019.622606:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332724224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017019.622620:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017019.622623:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017019.622625:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017019.622629:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1282 00000100:00080000:3.0:1734017019.622632:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734017019.622633:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017019.622635:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017019.622639:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017019.622647:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332724352/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017019.622653:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017019/real 1734017019] req@ffff8800a73cea00 x1818247332723712/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017019.622655:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017019.622658:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017019.622659:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017019.622663:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1282 00000100:00080000:3.0:1734017019.622665:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017019.622668:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017019.622672:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332723712/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017019.622672:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017019.622715:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332724480/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017019.622720:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017019.622723:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017019.622728:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332723712:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017019.622736:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332723840:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017019.622757:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017019.622765:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017019/real 1734017019] req@ffff8800a73ced80 x1818247332723840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017019.622776:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332723840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017019.622784:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017019.622787:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017019.622790:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332723840:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017019.622796:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332723968:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017019.622809:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017019.622815:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017019/real 1734017019] req@ffff8800a73cca80 x1818247332723968/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017019.622825:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332723968/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017019.622834:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017019.622837:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017019.622840:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332723968:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017019.622864:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332724096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017019.622876:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017019.622882:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017019/real 1734017019] req@ffff8800a73cd880 x1818247332724096/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017019.622892:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332724096/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017019.622900:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017019.622902:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017019.622905:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332724096:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017019.622910:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332724224:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017019.622921:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017019.622927:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017019/real 1734017019] req@ffff8800a73cfb80 x1818247332724224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017019.622937:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332724224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017019.622944:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017019.622946:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017019.622949:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332724224:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017019.622955:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332724352:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017019.622967:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017019.622973:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017019/real 1734017019] req@ffff8800a73cd500 x1818247332724352/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017019.622983:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332724352/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017019.622990:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017019.622992:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017019.622995:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332724352:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017019.623000:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332724480:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017019.623012:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017019.623018:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017019/real 1734017019] req@ffff8800a73cf800 x1818247332724480/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017019.623027:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332724480/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017019.623034:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017019.623036:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017019.623039:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332724480:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017019.746458:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017020.748479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017021.750490:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017022.752502:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017023.754493:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017024.630365:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017024.630375:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017024.630380:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017024.630391:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1287 00000100:00080000:3.0:1734017024.630401:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017024.630409:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017024.630447:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332724608/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017024.630480:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017024.630485:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017024.630487:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017024.630491:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1287 00000100:00080000:3.0:1734017024.630495:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017024.630499:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017024.630509:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017024.630521:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332724736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017024.630531:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017024.630534:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017024.630536:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017024.630540:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1287 00000100:00080000:3.0:1734017024.630542:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017024.630546:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017024.630550:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017024.630558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332724864/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017024.630567:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017024.630571:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017024.630572:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017024.630576:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1287 00000100:00080000:3.0:1734017024.630578:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017024.630582:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017024.630586:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332724608:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017024.630586:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017024.630594:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332724992/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017024.630648:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017024.630651:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017024.630653:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017024.630656:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1287 00000100:00080000:3.0:1734017024.630659:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017024.630663:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017024.630668:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017024.630677:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332725120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017024.630685:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017024.630688:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1734017024.630689:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017024.630690:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017024.630694:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1287 00000100:00080000:3.0:1734017024.630697:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017024.630700:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017024.630705:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017024.630711:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017024/real 1734017024] req@ffff8800a73cf480 x1818247332724608/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017024.630713:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332725248/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017024.630721:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017024.630725:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017024.630726:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017024.630729:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1287 00000100:00100000:0.0:1734017024.630731:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332724608/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017024.630732:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017024.630735:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017024.630739:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1734017024.630743:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017024.630746:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017024.630747:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332725376/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017024.630751:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332724608:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017024.630759:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332724736:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017024.630777:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017024.630784:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017024/real 1734017024] req@ffff8800a73cf800 x1818247332724736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017024.630796:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332724736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017024.630803:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017024.630806:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017024.630809:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332724736:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017024.630815:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332724864:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017024.630829:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017024.630835:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017024/real 1734017024] req@ffff8800a73cd500 x1818247332724864/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017024.630845:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332724864/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017024.630852:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017024.630856:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017024.630859:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332724864:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017024.630883:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332724992:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017024.630896:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017024.630902:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017024/real 1734017024] req@ffff8800a73cfb80 x1818247332724992/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017024.630912:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332724992/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017024.630919:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017024.630922:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017024.630925:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332724992:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017024.630929:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332725120:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017024.630941:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017024.630947:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017024/real 1734017024] req@ffff8800a73cd880 x1818247332725120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017024.630956:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332725120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017024.630964:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017024.630966:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017024.630969:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332725120:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017024.630974:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332725248:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017024.630989:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017024.630995:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017024/real 1734017024] req@ffff8800a73cca80 x1818247332725248/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017024.631005:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332725248/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017024.631012:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017024.631014:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017024.631017:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332725248:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017024.631022:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332725376:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017024.631034:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017024.631039:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017024/real 1734017024] req@ffff8800a73ced80 x1818247332725376/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017024.631049:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332725376/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017024.631056:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017024.631058:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017024.631061:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332725376:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017024.756468:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017025.758457:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017026.760479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017027.762492:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017028.764479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017029.638345:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017029.638355:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017029.638359:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017029.638370:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1292 00000100:00080000:3.0:1734017029.638380:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017029.638387:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017029.638423:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332725504/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017029.638456:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017029.638463:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017029.638465:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017029.638470:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1292 00000100:00080000:3.0:1734017029.638473:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017029.638477:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017029.638487:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017029.638498:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332725632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017029.638508:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017029.638543:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017029.638545:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017029.638549:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1292 00000100:00080000:3.0:1734017029.638552:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017029.638556:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017029.638561:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017029.638571:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332725760/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017029.638582:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017029.638585:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017029.638587:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017029.638591:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1292 00000100:00080000:3.0:1734017029.638593:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017029.638595:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332725504:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017029.638597:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017029.638602:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017029.638610:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332725888/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017029.638619:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017029.638622:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017029.638623:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017029.638626:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1292 00000100:00080000:3.0:1734017029.638629:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017029.638632:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017029.638636:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017029.638644:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332726016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017029.638652:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017029.638655:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017029.638657:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017029.638661:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1292 00000800:00000100:0.0:1734017029.638663:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017029.638663:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017029.638667:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017029.638671:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017029.638679:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332726144/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017029.638683:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017029/real 1734017029] req@ffff8800a73cea00 x1818247332725504/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017029.638686:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017029.638690:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017029.638692:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017029.638696:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1292 00000100:00080000:3.0:1734017029.638699:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017029.638702:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017029.638706:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017029.638710:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332725504/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017029.638716:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332726272/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017029.638720:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017029.638724:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017029.638728:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332725504:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017029.638736:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332725632:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017029.638753:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017029.638760:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017029/real 1734017029] req@ffff8800a73cdf80 x1818247332725632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017029.638771:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332725632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017029.638779:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017029.638781:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017029.638785:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332725632:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017029.638808:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332725760:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017029.638821:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017029.638827:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017029/real 1734017029] req@ffff8800a73cc700 x1818247332725760/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017029.638837:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332725760/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017029.638845:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017029.638848:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017029.638851:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332725760:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017029.638857:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332725888:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017029.638869:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017029.638875:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017029/real 1734017029] req@ffff8800a73ce680 x1818247332725888/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017029.638884:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332725888/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017029.638892:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017029.638894:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017029.638897:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332725888:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017029.638902:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332726016:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017029.638914:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017029.638920:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017029/real 1734017029] req@ffff8800a73cdc00 x1818247332726016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017029.638930:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332726016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017029.638938:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017029.638940:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017029.638943:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332726016:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017029.638948:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332726144:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017029.638963:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017029.638969:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017029/real 1734017029] req@ffff8800a73cf100 x1818247332726144/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017029.638979:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332726144/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017029.638986:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017029.638988:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017029.638991:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332726144:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017029.638996:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332726272:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017029.639007:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017029.639013:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017029/real 1734017029] req@ffff8800a73cce00 x1818247332726272/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017029.639022:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332726272/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017029.639030:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017029.639032:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017029.639035:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332726272:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017029.766478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017030.768541:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017031.770560:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017032.772445:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017033.774528:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017034.646386:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017034.646396:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017034.646401:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017034.646412:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1297 00000100:00080000:3.0:1734017034.646421:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017034.646464:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017034.646502:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332726400/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017034.646533:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017034.646538:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017034.646540:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017034.646544:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1297 00000100:00080000:3.0:1734017034.646548:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017034.646552:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017034.646561:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017034.646574:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332726528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017034.646585:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017034.646588:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017034.646590:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017034.646593:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1297 00000100:00080000:3.0:1734017034.646596:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017034.646599:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017034.646604:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017034.646612:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332726656/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017034.646620:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017034.646623:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017034.646625:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017034.646628:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1297 00000100:00080000:3.0:1734017034.646631:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017034.646634:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017034.646639:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017034.646640:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332726400:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017034.646646:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332726784/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017034.646655:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017034.646659:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017034.646660:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017034.646664:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1297 00000100:00080000:3.0:1734017034.646667:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017034.646670:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017034.646675:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017034.646683:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332726912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017034.646691:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017034.646694:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017034.646696:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017034.646699:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1297 00000100:00080000:3.0:1734017034.646702:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734017034.646705:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017034.646706:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017034.646710:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017034.646717:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332727040/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017034.646725:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017034.646726:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017034/real 1734017034] req@ffff8800a73cc380 x1818247332726400/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017034.646728:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017034.646729:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017034.646733:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1297 00000100:00080000:3.0:1734017034.646735:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017034.646739:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017034.646743:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017034.646751:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332727168/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017034.646754:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332726400/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017034.646764:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017034.646767:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017034.646772:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332726400:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017034.646779:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332726528:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017034.646796:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017034.646803:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017034/real 1734017034] req@ffff8800a73cce00 x1818247332726528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017034.646814:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332726528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017034.646821:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017034.646824:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017034.646827:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332726528:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017034.646833:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332726656:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017034.646847:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017034.646852:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017034/real 1734017034] req@ffff8800a73cf100 x1818247332726656/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017034.646863:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332726656/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017034.646870:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017034.646873:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017034.646876:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332726656:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017034.646902:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332726784:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017034.646914:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017034.646920:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017034/real 1734017034] req@ffff8800a73cdc00 x1818247332726784/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017034.646930:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332726784/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017034.646937:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017034.646940:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017034.646943:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332726784:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017034.646947:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332726912:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017034.646959:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017034.646965:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017034/real 1734017034] req@ffff8800a73ce680 x1818247332726912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017034.646975:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332726912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017034.646982:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017034.646984:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017034.646987:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332726912:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017034.646992:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332727040:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017034.647008:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017034.647013:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017034/real 1734017034] req@ffff8800a73cc700 x1818247332727040/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017034.647024:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332727040/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017034.647031:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017034.647033:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017034.647036:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332727040:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017034.647041:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332727168:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017034.647052:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017034.647058:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017034/real 1734017034] req@ffff8800a73cdf80 x1818247332727168/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017034.647067:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332727168/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017034.647074:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017034.647077:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017034.647079:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332727168:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017034.776508:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017035.778487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017036.780560:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017037.782518:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017038.784533:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017039.654404:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017039.654414:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017039.654418:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017039.654428:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1302 00000100:00080000:3.0:1734017039.654438:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017039.654445:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017039.654480:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332727296/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017039.654512:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017039.654516:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017039.654518:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017039.654522:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1302 00000100:00080000:3.0:1734017039.654525:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017039.654529:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017039.654539:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017039.654550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332727424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017039.654560:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017039.654563:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017039.654565:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017039.654568:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1302 00000100:00080000:3.0:1734017039.654572:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017039.654576:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017039.654580:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017039.654588:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332727552/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017039.654597:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017039.654600:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017039.654601:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017039.654605:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1302 00000100:00080000:3.0:1734017039.654607:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017039.654611:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017039.654615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017039.654616:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332727296:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017039.654622:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332727680/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017039.654631:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017039.654635:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017039.654636:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017039.654640:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1302 00000100:00080000:3.0:1734017039.654642:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017039.654645:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017039.654650:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017039.654657:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332727808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017039.654672:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017039.654675:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017039.654677:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1734017039.654679:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017039.654681:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1302 00000100:00080000:3.0:1734017039.654684:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017039.654687:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017039.654691:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017039.654698:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332727936/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017039.654700:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017039/real 1734017039] req@ffff8800a73cea00 x1818247332727296/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017039.654706:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017039.654709:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017039.654711:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017039.654714:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1302 00000100:00080000:3.0:1734017039.654717:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017039.654720:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017039.654724:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017039.654728:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332727296/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017039.654733:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332728064/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017039.654739:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017039.654742:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017039.654748:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332727296:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017039.654755:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332727424:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017039.654772:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017039.654779:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017039/real 1734017039] req@ffff8800a73ced80 x1818247332727424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017039.654790:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332727424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017039.654797:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017039.654800:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017039.654803:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332727424:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017039.654809:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332727552:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017039.654822:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017039.654828:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017039/real 1734017039] req@ffff8800a73cca80 x1818247332727552/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017039.654838:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332727552/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017039.654845:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017039.654848:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017039.654851:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332727552:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017039.654876:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332727680:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017039.654888:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017039.654894:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017039/real 1734017039] req@ffff8800a73cd880 x1818247332727680/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017039.654904:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332727680/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017039.654912:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017039.654914:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017039.654917:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332727680:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017039.654922:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332727808:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017039.654933:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017039.654939:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017039/real 1734017039] req@ffff8800a73cfb80 x1818247332727808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017039.654949:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332727808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017039.654956:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017039.654958:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017039.654961:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332727808:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017039.654966:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332727936:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017039.654981:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017039.654987:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017039/real 1734017039] req@ffff8800a73cd500 x1818247332727936/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017039.654997:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332727936/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017039.655004:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017039.655006:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017039.655009:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332727936:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017039.655014:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332728064:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017039.655026:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017039.655031:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017039/real 1734017039] req@ffff8800a73cf800 x1818247332728064/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017039.655041:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332728064/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017039.655049:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017039.655051:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017039.655054:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332728064:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017039.786502:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017040.788516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017041.790534:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017042.792495:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017043.794455:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017044.662367:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017044.662377:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017044.662382:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017044.662393:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1307 00000100:00080000:3.0:1734017044.662402:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017044.662410:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017044.662447:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332728192/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017044.662479:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017044.662484:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017044.662486:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017044.662490:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1307 00000100:00080000:3.0:1734017044.662493:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017044.662497:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017044.662506:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017044.662517:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332728320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017044.662527:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017044.662530:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017044.662532:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017044.662536:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1307 00000100:00080000:3.0:1734017044.662539:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017044.662542:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017044.662547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017044.662555:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332728448/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017044.662563:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017044.662567:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017044.662568:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017044.662572:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1307 00000100:00080000:3.0:1734017044.662574:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017044.662578:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017044.662582:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017044.662589:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332728192:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017044.662590:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332728576/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017044.662599:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017044.662602:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017044.662603:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017044.662606:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1307 00000100:00080000:3.0:1734017044.662609:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017044.662612:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017044.662616:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017044.662624:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332728704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017044.662632:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017044.662635:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017044.662636:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017044.662640:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1307 00000100:00080000:3.0:1734017044.662643:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017044.662646:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017044.662652:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017044.662653:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017044.662659:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332728832/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017044.662668:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017044.662671:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017044.662672:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017044.662674:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017044/real 1734017044] req@ffff8800a73cf480 x1818247332728192/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017044.662676:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1307 00000100:00080000:3.0:1734017044.662678:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017044.662682:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017044.662686:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017044.662693:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332728960/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017044.662701:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332728192/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017044.662711:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017044.662714:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017044.662719:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332728192:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017044.662726:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332728320:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017044.662743:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017044.662750:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017044/real 1734017044] req@ffff8800a73cf800 x1818247332728320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017044.662760:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332728320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017044.662768:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017044.662771:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017044.662774:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332728320:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017044.662780:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332728448:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017044.662793:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017044.662799:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017044/real 1734017044] req@ffff8800a73cd500 x1818247332728448/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017044.662809:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332728448/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017044.662817:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017044.662820:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017044.662823:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332728448:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017044.662846:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332728576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017044.662859:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017044.662865:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017044/real 1734017044] req@ffff8800a73cfb80 x1818247332728576/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017044.662875:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332728576/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017044.662882:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017044.662884:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017044.662887:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332728576:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017044.662892:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332728704:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017044.662904:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017044.662909:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017044/real 1734017044] req@ffff8800a73cd880 x1818247332728704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017044.662919:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332728704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017044.662926:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017044.662929:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017044.662932:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332728704:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017044.662937:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332728832:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017044.662951:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017044.662957:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017044/real 1734017044] req@ffff8800a73cca80 x1818247332728832/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017044.662967:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332728832/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017044.662974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017044.662976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017044.662979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332728832:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017044.662984:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332728960:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017044.662996:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017044.663001:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017044/real 1734017044] req@ffff8800a73ced80 x1818247332728960/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017044.663011:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332728960/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017044.663018:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017044.663020:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017044.663023:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332728960:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017044.796488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017045.798462:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017046.800514:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017047.802493:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017048.804476:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017049.670344:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017049.670354:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017049.670359:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017049.670370:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1312 00000100:00080000:3.0:1734017049.670380:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017049.670388:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017049.670424:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332729088/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017049.670459:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017049.670466:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017049.670468:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017049.670473:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1312 00000100:00080000:3.0:1734017049.670476:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017049.670480:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017049.670490:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017049.670500:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332729216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017049.670511:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017049.670521:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017049.670523:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017049.670527:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1312 00000100:00080000:3.0:1734017049.670529:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017049.670533:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017049.670538:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017049.670546:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332729344/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017049.670554:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017049.670557:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017049.670559:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017049.670562:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1312 00000100:00080000:3.0:1734017049.670565:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017049.670568:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017049.670573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017049.670581:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332729088:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017049.670581:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332729472/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017049.670590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017049.670593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017049.670595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017049.670598:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1312 00000100:00080000:3.0:1734017049.670601:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017049.670604:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017049.670608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017049.670617:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332729600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017049.670625:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017049.670628:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017049.670629:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017049.670633:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1312 00000100:00080000:3.0:1734017049.670637:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017049.670640:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017049.670644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017049.670647:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017049.670652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332729728/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017049.670660:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017049.670664:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017049.670666:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017049.670669:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017049/real 1734017049] req@ffff8800a73cea00 x1818247332729088/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017049.670670:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1312 00000100:00080000:3.0:1734017049.670673:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017049.670676:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017049.670680:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017049.670689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332729856/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017049.670696:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332729088/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017049.670707:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017049.670710:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017049.670715:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332729088:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017049.670722:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332729216:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017049.670739:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017049.670746:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017049/real 1734017049] req@ffff8800a73cdf80 x1818247332729216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017049.670757:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332729216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017049.670764:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017049.670767:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017049.670770:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332729216:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017049.670776:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332729344:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017049.670789:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017049.670795:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017049/real 1734017049] req@ffff8800a73cc700 x1818247332729344/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017049.670805:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332729344/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017049.670812:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017049.670815:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017049.670819:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332729344:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017049.670844:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332729472:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017049.670857:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017049.670863:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017049/real 1734017049] req@ffff8800a73ce680 x1818247332729472/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017049.670873:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332729472/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017049.670880:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017049.670883:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017049.670886:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332729472:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017049.670890:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332729600:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017049.670902:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017049.670908:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017049/real 1734017049] req@ffff8800a73cdc00 x1818247332729600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017049.670918:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332729600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017049.670925:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017049.670927:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017049.670930:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332729600:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017049.670935:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332729728:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017049.670950:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017049.670956:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017049/real 1734017049] req@ffff8800a73cf100 x1818247332729728/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017049.670967:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332729728/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017049.670974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017049.670976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017049.670978:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332729728:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017049.670983:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332729856:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017049.670995:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017049.671001:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017049/real 1734017049] req@ffff8800a73cce00 x1818247332729856/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017049.671011:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332729856/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017049.671019:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017049.671021:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017049.671024:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332729856:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017049.806443:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017050.808504:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017051.810498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017052.812521:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017053.814500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017054.678344:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017054.678358:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017054.678364:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017054.678378:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1317 00000100:00080000:3.0:1734017054.678391:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017054.678402:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017054.678441:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332729984/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017054.678481:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017054.678486:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017054.678488:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017054.678493:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1317 00000100:00080000:3.0:1734017054.678496:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017054.678500:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017054.678510:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017054.678523:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332730112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017054.678535:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017054.678538:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017054.678540:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017054.678543:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1317 00000100:00080000:3.0:1734017054.678546:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017054.678549:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017054.678554:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017054.678562:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332730240/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017054.678571:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017054.678574:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017054.678575:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017054.678579:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1317 00000100:00080000:3.0:1734017054.678581:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017054.678585:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017054.678589:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017054.678597:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332730368/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017054.678606:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017054.678609:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017054.678611:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017054.678614:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332729984:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017054.678615:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1317 00000100:00080000:3.0:1734017054.678618:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017054.678621:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017054.678626:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017054.678635:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332730496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017054.678643:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017054.678646:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017054.678647:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017054.678651:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1317 00000100:00080000:3.0:1734017054.678654:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017054.678658:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017054.678662:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017054.678670:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332730624/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017054.678678:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017054.678681:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017054.678682:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017054.678685:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1317 00000100:00080000:3.0:1734017054.678689:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017054.678693:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017054.678697:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017054.678704:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332730752/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734017054.678710:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017054.678738:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017054/real 1734017054] req@ffff8800a73cc380 x1818247332729984/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017054.678769:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332729984/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017054.678781:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017054.678785:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017054.678791:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332729984:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017054.678799:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332730112:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017054.678819:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017054.678826:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017054/real 1734017054] req@ffff8800a73cce00 x1818247332730112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017054.678837:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332730112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017054.678845:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017054.678848:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017054.678851:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332730112:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017054.678858:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332730240:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017054.678873:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017054.678879:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017054/real 1734017054] req@ffff8800a73cf100 x1818247332730240/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017054.678889:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332730240/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017054.678896:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017054.678900:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017054.678903:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332730240:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017054.678934:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332730368:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017054.678948:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017054.678954:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017054/real 1734017054] req@ffff8800a73cdc00 x1818247332730368/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017054.678965:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332730368/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017054.678973:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017054.678976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017054.678978:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332730368:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017054.678984:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332730496:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017054.678997:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017054.679002:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017054/real 1734017054] req@ffff8800a73ce680 x1818247332730496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017054.679012:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332730496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017054.679020:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017054.679022:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017054.679025:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332730496:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017054.679032:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332730624:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017054.679048:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017054.679055:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017054/real 1734017054] req@ffff8800a73cc700 x1818247332730624/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017054.679065:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332730624/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017054.679072:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017054.679075:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017054.679077:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332730624:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017054.679120:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332730752:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017054.679140:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017054.679147:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017054/real 1734017054] req@ffff8800a73cdf80 x1818247332730752/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017054.679158:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332730752/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017054.679167:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017054.679169:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017054.679172:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332730752:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017054.816499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017055.818479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017056.820507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017057.822448:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017058.824499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017059.686354:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017059.686364:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017059.686369:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017059.686380:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1322 00000100:00080000:3.0:1734017059.686391:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017059.686399:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017059.686435:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332730880/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017059.686471:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017059.686475:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017059.686477:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017059.686481:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1322 00000100:00080000:3.0:1734017059.686485:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017059.686489:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017059.686498:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017059.686509:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332731008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017059.686519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017059.686522:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017059.686524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017059.686528:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1322 00000100:00080000:3.0:1734017059.686531:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017059.686535:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017059.686540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017059.686548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332731136/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017059.686557:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017059.686560:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017059.686562:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017059.686565:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1322 00000100:00080000:3.0:1734017059.686568:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017059.686571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017059.686575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017059.686582:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332731264/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017059.686592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017059.686595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017059.686597:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017059.686599:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332730880:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017059.686601:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1322 00000100:00080000:3.0:1734017059.686603:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017059.686607:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017059.686611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017059.686619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332731392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017059.686634:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017059.686637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017059.686639:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017059.686643:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1322 00000100:00080000:3.0:1734017059.686646:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017059.686650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017059.686654:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017059.686661:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332731520/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017059.686669:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017059.686672:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017059.686674:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017059.686677:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1322 00000100:00080000:3.0:1734017059.686680:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017059.686683:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017059.686687:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017059.686688:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017059.686696:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332731648/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017059.686715:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017059/real 1734017059] req@ffff8800a73cea00 x1818247332730880/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017059.686746:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332730880/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017059.686759:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017059.686763:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017059.686768:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332730880:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017059.686776:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332731008:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017059.686794:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017059.686801:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017059/real 1734017059] req@ffff8800a73ced80 x1818247332731008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017059.686812:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332731008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017059.686820:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017059.686823:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017059.686826:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332731008:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017059.686833:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332731136:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017059.686848:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017059.686854:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017059/real 1734017059] req@ffff8800a73cca80 x1818247332731136/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017059.686864:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332731136/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017059.686871:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017059.686875:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017059.686878:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332731136:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017059.686909:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332731264:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017059.686923:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017059.686929:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017059/real 1734017059] req@ffff8800a73cd880 x1818247332731264/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017059.686940:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332731264/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017059.686947:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017059.686950:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017059.686953:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332731264:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017059.686958:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332731392:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017059.686971:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017059.686977:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017059/real 1734017059] req@ffff8800a73cfb80 x1818247332731392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017059.686987:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332731392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017059.686994:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017059.686997:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017059.687042:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332731392:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017059.687050:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332731520:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017059.687071:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017059.687078:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017059/real 1734017059] req@ffff8800a73cd500 x1818247332731520/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017059.687089:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332731520/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017059.687097:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017059.687100:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017059.687103:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332731520:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017059.687109:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332731648:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017059.687122:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017059.687128:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017059/real 1734017059] req@ffff8800a73cf800 x1818247332731648/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017059.687138:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332731648/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017059.687145:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017059.687147:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017059.687151:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332731648:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017059.826455:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017060.828512:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017061.830458:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017062.832476:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017063.834491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017064.694334:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017064.694344:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017064.694349:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017064.694360:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1327 00000100:00080000:3.0:1734017064.694370:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017064.694378:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017064.694414:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332731776/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017064.694450:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017064.694456:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017064.694459:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017064.694465:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1327 00000100:00080000:3.0:1734017064.694470:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017064.694476:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017064.694489:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017064.694509:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332731904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017064.694516:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332731776:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017064.694525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017064.694529:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017064.694531:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017064.694537:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1327 00000100:00080000:3.0:1734017064.694541:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017064.694545:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017064.694551:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017064.694564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332732032/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017064.694576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017064.694582:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017064.694584:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1734017064.694587:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017064.694589:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1327 00000100:00080000:3.0:1734017064.694594:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017064.694599:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017064.694605:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017064.694609:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017064/real 1734017064] req@ffff8800a73cf480 x1818247332731776/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017064.694618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332732160/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017064.694632:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017064.694637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017064.694639:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017064.694644:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332731776/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017064.694645:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1327 00000100:00080000:3.0:1734017064.694649:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017064.694655:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:0.0:1734017064.694658:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1734017064.694661:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1734017064.694662:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017064.694669:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332731776:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017064.694673:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332732288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017064.694686:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017064.694690:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017064.694693:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017064.694696:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332731904:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017064.694699:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1327 00000100:00080000:3.0:1734017064.694703:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017064.694708:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017064.694716:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017064.694722:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017064.694729:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332732416/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017064.694732:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017064/real 1734017064] req@ffff8800a73cf800 x1818247332731904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017064.694744:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017064.694749:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332731904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017064.694750:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017064.694752:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017064.694758:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1327 00000100:00080000:0.0:1734017064.694762:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017064.694762:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017064.694766:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734017064.694767:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017064.694772:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332731904:192.168.201.152@tcp:38:kworker.0 00000008:00100000:3.0:1734017064.694773:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017064.694780:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332732032:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734017064.694785:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332732544/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734017064.694802:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017064.694811:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017064/real 1734017064] req@ffff8800a73cd500 x1818247332732032/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017064.694828:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332732032/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017064.694841:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017064.694845:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017064.694851:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332732032:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017064.694859:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332732160:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017064.694885:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017064.694894:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017064/real 1734017064] req@ffff8800a73cfb80 x1818247332732160/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017064.694905:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332732160/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017064.694964:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017064.694968:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017064.694972:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332732160:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017064.694978:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332732288:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017064.694997:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017064.695005:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017064/real 1734017064] req@ffff8800a73cd880 x1818247332732288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017064.695016:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332732288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017064.695024:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017064.695027:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017064.695030:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332732288:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017064.695047:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332732416:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017064.695063:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017064.695069:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017064/real 1734017064] req@ffff8800a73cca80 x1818247332732416/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017064.695080:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332732416/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017064.695087:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017064.695089:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017064.695092:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332732416:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017064.695097:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332732544:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017064.695109:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017064.695114:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017064/real 1734017064] req@ffff8800a73ced80 x1818247332732544/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017064.695124:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332732544/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017064.695131:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017064.695133:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017064.695136:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332732544:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017064.836423:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017065.838494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017066.840481:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017067.842502:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017068.844504:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017069.702359:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017069.702368:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017069.702373:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017069.702384:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1332 00000100:00080000:3.0:1734017069.702393:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017069.702401:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017069.702438:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332732672/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017069.702472:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017069.702479:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017069.702481:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017069.702485:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1332 00000100:00080000:3.0:1734017069.702489:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017069.702493:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017069.702502:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017069.702513:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332732800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017069.702523:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017069.702526:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017069.702528:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017069.702532:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1332 00000100:00080000:3.0:1734017069.702535:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017069.702538:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017069.702543:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017069.702551:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332732928/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017069.702560:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017069.702563:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017069.702564:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017069.702567:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1332 00000100:00080000:3.0:1734017069.702570:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017069.702573:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017069.702578:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332732672:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017069.702579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017069.702586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332733056/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017069.702596:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017069.702599:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017069.702600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017069.702604:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1332 00000100:00080000:3.0:1734017069.702606:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017069.702610:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017069.702614:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017069.702621:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332733184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017069.702629:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017069.702632:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017069.702634:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017069.702638:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1332 00000100:00080000:3.0:1734017069.702640:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734017069.702644:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017069.702644:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017069.702648:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017069.702656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332733312/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017069.702664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017069.702665:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017069/real 1734017069] req@ffff8800a73cea00 x1818247332732672/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017069.702667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017069.702670:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017069.702673:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1332 00000100:00080000:3.0:1734017069.702676:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017069.702679:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017069.702684:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017069.702691:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332732672/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017069.702692:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332733440/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017069.702701:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017069.702704:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017069.702709:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332732672:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017069.702716:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332732800:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017069.702734:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017069.702740:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017069/real 1734017069] req@ffff8800a73cdf80 x1818247332732800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017069.702752:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332732800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017069.702759:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017069.702762:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017069.702765:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332732800:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017069.702771:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332732928:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017069.702785:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017069.702791:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017069/real 1734017069] req@ffff8800a73cc700 x1818247332732928/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017069.702801:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332732928/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017069.702808:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017069.702812:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017069.702815:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332732928:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017069.702877:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332733056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017069.702895:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017069.702903:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017069/real 1734017069] req@ffff8800a73ce680 x1818247332733056/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017069.702914:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332733056/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017069.702921:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017069.702923:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017069.702926:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332733056:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017069.702931:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332733184:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017069.702943:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017069.702949:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017069/real 1734017069] req@ffff8800a73cdc00 x1818247332733184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017069.702959:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332733184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017069.702967:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017069.702969:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017069.702972:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332733184:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017069.702977:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332733312:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017069.702992:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017069.702998:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017069/real 1734017069] req@ffff8800a73cf100 x1818247332733312/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017069.703009:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332733312/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017069.703016:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017069.703018:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017069.703021:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332733312:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017069.703026:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332733440:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017069.703037:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017069.703043:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017069/real 1734017069] req@ffff8800a73cce00 x1818247332733440/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017069.703053:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332733440/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017069.703061:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017069.703063:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017069.703066:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332733440:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017069.846404:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017070.848457:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017071.850503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017072.852498:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017073.854460:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017074.710359:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017074.710369:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017074.710374:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017074.710385:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1337 00000100:00080000:3.0:1734017074.710395:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017074.710403:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017074.710440:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332733568/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017074.710474:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017074.710478:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017074.710480:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017074.710485:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1337 00000100:00080000:3.0:1734017074.710488:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017074.710492:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017074.710501:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017074.710512:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332733696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017074.710524:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017074.710527:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017074.710529:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017074.710533:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1337 00000100:00080000:3.0:1734017074.710535:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017074.710538:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017074.710543:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017074.710551:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332733824/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017074.710560:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017074.710562:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017074.710564:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017074.710567:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1337 00000100:00080000:3.0:1734017074.710570:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017074.710573:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017074.710577:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017074.710581:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332733568:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017074.710586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332733952/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017074.710594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017074.710597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017074.710599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017074.710602:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1337 00000100:00080000:3.0:1734017074.710606:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017074.710609:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017074.710613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017074.710621:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332734080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017074.710629:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017074.710632:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017074.710634:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017074.710637:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1337 00000100:00080000:3.0:1734017074.710640:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017074.710644:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017074.710648:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017074.710650:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017074.710655:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332734208/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017074.710663:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017074.710666:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017074.710667:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017074.710671:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017074/real 1734017074] req@ffff8800a73cc380 x1818247332733568/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017074.710671:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1337 00000100:00080000:3.0:1734017074.710674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017074.710677:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017074.710681:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017074.710688:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332734336/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017074.710698:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332733568/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017074.710708:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017074.710712:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017074.710716:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332733568:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017074.710723:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332733696:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017074.710740:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017074.710793:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017074/real 1734017074] req@ffff8800a73cce00 x1818247332733696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017074.710805:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332733696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017074.710812:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017074.710815:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017074.710819:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332733696:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017074.710825:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332733824:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017074.710842:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017074.710848:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017074/real 1734017074] req@ffff8800a73cf100 x1818247332733824/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017074.710859:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332733824/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017074.710867:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017074.710869:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017074.710873:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332733824:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017074.710897:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332733952:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017074.710910:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017074.710916:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017074/real 1734017074] req@ffff8800a73cdc00 x1818247332733952/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017074.710926:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332733952/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017074.710933:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017074.710935:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017074.710938:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332733952:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017074.710943:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332734080:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017074.710954:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017074.710960:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017074/real 1734017074] req@ffff8800a73ce680 x1818247332734080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017074.710970:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332734080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017074.710977:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017074.710979:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017074.710983:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332734080:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017074.710988:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332734208:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017074.711003:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017074.711009:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017074/real 1734017074] req@ffff8800a73cc700 x1818247332734208/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017074.711019:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332734208/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017074.711026:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017074.711029:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017074.711032:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332734208:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017074.711037:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332734336:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017074.711048:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017074.711054:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017074/real 1734017074] req@ffff8800a73cdf80 x1818247332734336/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017074.711064:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332734336/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017074.711072:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017074.711074:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017074.711077:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332734336:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017074.856469:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017075.858493:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017076.860497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017077.862474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017078.864505:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017079.718379:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017079.718389:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017079.718394:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017079.718405:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1342 00000100:00080000:3.0:1734017079.718416:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017079.718424:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017079.718461:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332734464/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017079.718491:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017079.718495:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017079.718497:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017079.718501:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1342 00000100:00080000:3.0:1734017079.718504:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017079.718508:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017079.718518:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017079.718528:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332734592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017079.718538:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017079.718541:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017079.718543:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017079.718547:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1342 00000100:00080000:3.0:1734017079.718551:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017079.718554:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017079.718559:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017079.718567:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332734720/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017079.718576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017079.718579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017079.718580:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017079.718583:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1342 00000100:00080000:3.0:1734017079.718586:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017079.718589:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017079.718593:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017079.718600:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332734464:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017079.718601:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332734848/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017079.718610:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017079.718613:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017079.718614:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017079.718618:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1342 00000100:00080000:3.0:1734017079.718620:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017079.718624:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017079.718628:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017079.718636:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332734976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017079.718651:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017079.718654:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017079.718656:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017079.718700:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1342 00000100:00080000:3.0:1734017079.718703:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017079.718707:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017079.718712:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017079.718716:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332734592:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734017079.718721:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332735104/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017079.718730:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017079.718734:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017079.718736:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017079.718739:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1342 00000100:00100000:1.0:1734017079.718740:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332734720:192.168.201.152@tcp:8:kworker.0 00000100:00080000:3.0:1734017079.718742:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017079.718745:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017079.718750:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017079.718759:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332735232/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017079.718764:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332734848:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017079.718779:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332734976:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017079.718795:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332735104:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017079.718811:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332735232:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017079.866460:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017080.868460:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017081.870478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1734017082.724522:0:1958:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:2.0:1734017082.724576:0:1958:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1734017082.724597:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734017082.724672:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734017082.724681:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734017082.724688:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734017082.724694:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734017082.724701:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734017082.724706:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734017082.724791:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017079/real 1734017082] req@ffff8800a73cf800 x1818247332735232/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017082.724831:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332735232/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017082.724851:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017082.724857:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017082.724865:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332735232:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017082.724880:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017079/real 1734017082] req@ffff8800a73cd880 x1818247332734848/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017082.724900:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332734848/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017082.724912:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017082.724917:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017082.724924:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332734848:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017082.724936:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017079/real 1734017082] req@ffff8800a73cfb80 x1818247332734976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017082.724956:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332734976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017082.724969:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017082.724974:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017082.724979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332734976:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017082.724997:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017079/real 1734017082] req@ffff8800a73cd500 x1818247332735104/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017082.725016:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332735104/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017082.725029:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017082.725033:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017082.725039:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332735104:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017082.725050:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017079/real 1734017082] req@ffff8800a73cea00 x1818247332734464/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017082.725070:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332734464/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017082.725084:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017082.725088:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017082.725093:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332734464:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017082.725105:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017079/real 1734017082] req@ffff8800a73ced80 x1818247332734592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017082.725124:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332734592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017082.725137:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017082.725142:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017082.725148:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332734592:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017082.725159:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017079/real 1734017082] req@ffff8800a73cca80 x1818247332734720/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017082.725179:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332734720/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017082.725192:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017082.725197:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017082.725202:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332734720:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017082.872493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017083.874508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017084.726343:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017084.726353:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017084.726358:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017084.726369:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1347 00000100:00080000:3.0:1734017084.726379:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017084.726387:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017084.726423:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332735360/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017084.726458:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017084.726462:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017084.726464:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017084.726469:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1347 00000100:00080000:3.0:1734017084.726472:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017084.726476:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017084.726485:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017084.726498:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332735488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017084.726508:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017084.726511:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017084.726513:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017084.726517:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1347 00000100:00080000:3.0:1734017084.726520:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017084.726523:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017084.726528:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017084.726536:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332735616/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017084.726545:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017084.726549:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017084.726550:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017084.726554:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1347 00000100:00080000:3.0:1734017084.726557:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017084.726560:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017084.726564:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017084.726606:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332735744/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017084.726617:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017084.726620:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1734017084.726621:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332735360:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017084.726622:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017084.726626:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1347 00000100:00080000:3.0:1734017084.726629:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017084.726633:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017084.726638:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017084.726647:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332735872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017084.726656:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017084.726659:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017084.726661:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017084.726665:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1347 00000100:00080000:3.0:1734017084.726667:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017084.726670:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017084.726676:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017084.726683:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332736000/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017084.726692:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017084.726695:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017084.726696:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017084.726699:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1347 00000100:00080000:3.0:1734017084.726704:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017084.726707:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017084.726712:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017084.726718:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017084.726720:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332736128/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017084.726749:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017084/real 1734017084] req@ffff8800a73cf480 x1818247332735360/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017084.726771:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332735360/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017084.726784:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017084.726788:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017084.726793:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332735360:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017084.726802:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332735488:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017084.726821:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017084.726828:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017084/real 1734017084] req@ffff8800a73cca80 x1818247332735488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017084.726840:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332735488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017084.726856:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017084.726859:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017084.726863:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332735488:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017084.726869:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332735616:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017084.726884:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017084.726890:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017084/real 1734017084] req@ffff8800a73ced80 x1818247332735616/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017084.726901:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332735616/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017084.726908:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017084.726911:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017084.726914:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332735616:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017084.726945:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332735744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017084.726958:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017084.726964:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017084/real 1734017084] req@ffff8800a73cea00 x1818247332735744/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017084.726975:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332735744/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017084.726982:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017084.726985:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017084.726988:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332735744:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017084.726995:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332735872:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017084.727008:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017084.727015:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017084/real 1734017084] req@ffff8800a73cd500 x1818247332735872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017084.727025:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332735872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017084.727034:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017084.727036:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017084.727039:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332735872:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017084.727045:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332736000:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017084.727058:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017084.727064:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017084/real 1734017084] req@ffff8800a73cfb80 x1818247332736000/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017084.727073:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332736000/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017084.727080:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017084.727083:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017084.727086:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332736000:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017084.727092:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332736128:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017084.727104:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017084.727110:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017084/real 1734017084] req@ffff8800a73cd880 x1818247332736128/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017084.727121:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332736128/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017084.727129:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017084.727131:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017084.727134:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332736128:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017084.876480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017085.878519:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017086.880487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017087.882442:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017088.884492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017089.734376:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017089.734387:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017089.734391:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017089.734401:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1352 00000100:00080000:3.0:1734017089.734411:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017089.734419:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017089.734455:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332736256/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017089.734529:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017089.734536:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017089.734538:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017089.734543:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1352 00000100:00080000:3.0:1734017089.734547:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017089.734551:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017089.734562:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017089.734575:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332736384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017089.734586:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017089.734589:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017089.734591:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017089.734594:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332736256:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017089.734595:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1352 00000100:00080000:3.0:1734017089.734597:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017089.734601:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017089.734606:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017089.734614:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332736512/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017089.734622:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017089.734625:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017089.734627:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017089.734631:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1352 00000100:00080000:3.0:1734017089.734633:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017089.734637:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017089.734642:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017089.734650:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332736640/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017089.734658:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734017089.734661:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017089.734661:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017089.734663:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017089.734666:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1352 00000100:00080000:3.0:1734017089.734669:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017089.734673:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017089.734677:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734017089.734684:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017089/real 1734017089] req@ffff8800a73cf800 x1818247332736256/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017089.734685:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332736768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017089.734693:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017089.734696:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017089.734698:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017089.734702:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1352 00000100:00100000:1.0:1734017089.734704:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332736256/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017089.734704:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017089.734708:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017089.734712:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734017089.734715:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017089.734718:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017089.734719:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332736896/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017089.734723:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332736256:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017089.734727:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017089.734731:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017089.734733:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017089.734737:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1352 00000100:00080000:3.0:1734017089.734740:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017089.734743:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017089.734745:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332736384:192.168.201.152@tcp:38:kworker.0 00000008:00100000:3.0:1734017089.734747:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017089.734756:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332737024/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017089.734763:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017089.734770:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017089/real 1734017089] req@ffff8800a73cdf80 x1818247332736384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017089.734781:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332736384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017089.734790:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017089.734793:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017089.734796:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332736384:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017089.734802:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332736512:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017089.734815:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017089.734821:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017089/real 1734017089] req@ffff8800a73cc700 x1818247332736512/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017089.734831:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332736512/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017089.734838:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017089.734841:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017089.734844:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332736512:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017089.734849:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332736640:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017089.734860:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017089.734866:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017089/real 1734017089] req@ffff8800a73ce680 x1818247332736640/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017089.734876:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332736640/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017089.734883:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017089.734885:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017089.734888:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332736640:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017089.734894:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332736768:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017089.734907:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017089.734913:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017089/real 1734017089] req@ffff8800a73cdc00 x1818247332736768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017089.734922:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332736768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017089.734930:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017089.734932:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017089.734935:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332736768:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017089.734939:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332736896:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017089.734951:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017089.734957:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017089/real 1734017089] req@ffff8800a73cf100 x1818247332736896/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017089.734967:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332736896/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017089.734974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017089.734976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017089.734979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332736896:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017089.734995:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332737024:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017089.735008:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017089.735014:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017089/real 1734017089] req@ffff8800a73cce00 x1818247332737024/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017089.735024:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332737024/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017089.735032:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017089.735034:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017089.735037:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332737024:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017089.886469:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017090.888519:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017091.890526:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017092.892527:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017093.894485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017094.742290:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017094.742298:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017094.742303:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017094.742313:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1357 00000100:00080000:3.0:1734017094.742323:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017094.742331:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017094.742366:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332737152/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017094.742399:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017094.742432:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017094.742435:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017094.742440:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1357 00000100:00080000:3.0:1734017094.742443:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017094.742447:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017094.742457:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017094.742471:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332737280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017094.742475:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332737152:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017094.742483:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017094.742486:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017094.742488:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017094.742492:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1357 00000100:00080000:3.0:1734017094.742494:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017094.742498:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017094.742502:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017094.742511:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332737408/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017094.742520:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017094.742523:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017094.742524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017094.742528:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1357 00000100:00080000:3.0:1734017094.742530:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017094.742533:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017094.742538:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017094.742540:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017094.742545:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332737536/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017094.742553:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017094.742557:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017094.742558:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017094.742562:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1357 00000100:00100000:1.0:1734017094.742563:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017094/real 1734017094] req@ffff8800a73cc380 x1818247332737152/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017094.742565:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017094.742568:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017094.742573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017094.742581:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332737664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017094.742583:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332737152/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017094.742589:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017094.742592:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017094.742593:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734017094.742594:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017094.742597:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734017094.742598:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1357 00000100:00080000:3.0:1734017094.742600:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017094.742602:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332737152:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017094.742604:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017094.742608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017094.742615:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332737792/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017094.742623:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017094.742624:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332737280:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017094.742627:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017094.742628:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017094.742632:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1357 00000100:00080000:3.0:1734017094.742634:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017094.742637:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017094.742641:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017094.742642:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017094.742649:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332737920/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017094.742650:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017094/real 1734017094] req@ffff8800a73cce00 x1818247332737280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017094.742661:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332737280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017094.742670:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017094.742673:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017094.742676:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332737280:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017094.742683:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332737408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017094.742696:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017094.742702:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017094/real 1734017094] req@ffff8800a73cf100 x1818247332737408/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017094.742712:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332737408/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017094.742719:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017094.742722:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017094.742725:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332737408:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017094.742731:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332737536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017094.742743:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017094.742749:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017094/real 1734017094] req@ffff8800a73cdc00 x1818247332737536/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017094.742758:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332737536/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017094.742766:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017094.742768:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017094.742771:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332737536:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017094.742777:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332737664:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017094.742789:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017094.742795:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017094/real 1734017094] req@ffff8800a73ce680 x1818247332737664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017094.742805:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332737664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017094.742812:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017094.742815:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017094.742818:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332737664:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017094.742833:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332737792:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017094.742845:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017094.742851:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017094/real 1734017094] req@ffff8800a73cc700 x1818247332737792/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017094.742861:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332737792/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017094.742869:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017094.742871:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017094.742874:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332737792:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017094.742879:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332737920:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017094.742891:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017094.742897:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017094/real 1734017094] req@ffff8800a73cdf80 x1818247332737920/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017094.742908:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332737920/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017094.742916:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017094.742918:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017094.742921:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332737920:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017094.896470:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017095.898489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017096.900559:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017097.902511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017098.904535:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017099.750385:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017099.750394:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017099.750398:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017099.750408:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1362 00000100:00080000:3.0:1734017099.750418:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017099.750426:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017099.750463:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332738048/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017099.750495:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017099.750499:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017099.750501:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017099.750505:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1362 00000100:00080000:3.0:1734017099.750508:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017099.750512:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017099.750522:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017099.750533:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332738176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017099.750543:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017099.750546:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017099.750548:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017099.750552:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1362 00000100:00080000:3.0:1734017099.750556:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017099.750559:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017099.750569:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017099.750577:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332738304/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017099.750585:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017099.750588:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017099.750590:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017099.750593:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1362 00000100:00080000:3.0:1734017099.750595:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017099.750599:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017099.750603:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017099.750607:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332738048:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017099.750611:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332738432/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017099.750619:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017099.750623:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017099.750624:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017099.750627:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1362 00000100:00080000:3.0:1734017099.750630:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017099.750633:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017099.750637:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017099.750645:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332738560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017099.750660:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017099.750663:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017099.750665:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017099.750668:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1362 00000100:00080000:3.0:1734017099.750671:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017099.750674:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734017099.750676:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017099.750679:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017099.750685:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332738688/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017099.750693:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017099.750697:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017099.750698:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017099.750699:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017099/real 1734017099] req@ffff8800a73cf800 x1818247332738048/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017099.750701:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1362 00000100:00080000:3.0:1734017099.750704:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017099.750707:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017099.750711:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017099.750718:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332738816/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017099.750719:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332738048/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017099.750730:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017099.750734:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017099.750738:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332738048:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017099.750746:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332738176:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017099.750763:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017099.750770:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017099/real 1734017099] req@ffff8800a73cd880 x1818247332738176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017099.750780:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332738176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017099.750789:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017099.750792:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017099.750795:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332738176:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017099.750801:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332738304:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017099.750814:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017099.750820:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017099/real 1734017099] req@ffff8800a73cfb80 x1818247332738304/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017099.750830:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332738304/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017099.750837:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017099.750840:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017099.750843:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332738304:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017099.750868:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332738432:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017099.750881:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017099.750887:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017099/real 1734017099] req@ffff8800a73cd500 x1818247332738432/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017099.750897:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332738432/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017099.750905:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017099.750907:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017099.750910:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332738432:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017099.750916:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332738560:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017099.750928:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017099.750934:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017099/real 1734017099] req@ffff8800a73cea00 x1818247332738560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017099.750944:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332738560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017099.750951:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017099.750953:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017099.750956:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332738560:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017099.750961:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332738688:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017099.750972:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017099.750978:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017099/real 1734017099] req@ffff8800a73ced80 x1818247332738688/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017099.750988:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332738688/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017099.750995:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017099.750997:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017099.751000:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332738688:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017099.751006:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332738816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017099.751018:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017099.751024:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017099/real 1734017099] req@ffff8800a73cca80 x1818247332738816/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017099.751034:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332738816/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017099.751042:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017099.751044:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017099.751047:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332738816:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017099.906510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017100.908528:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017101.910524:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017102.912522:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017103.914480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017104.758342:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017104.758351:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017104.758356:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017104.758366:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1367 00000100:00080000:3.0:1734017104.758376:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017104.758384:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017104.758420:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332738944/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017104.758452:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017104.758456:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017104.758459:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017104.758463:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1367 00000100:00080000:3.0:1734017104.758466:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017104.758470:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017104.758480:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017104.758492:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332739072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017104.758502:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017104.758505:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017104.758507:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017104.758511:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1367 00000100:00080000:3.0:1734017104.758513:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017104.758517:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017104.758522:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017104.758530:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332739200/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017104.758538:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017104.758542:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017104.758543:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017104.758547:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1367 00000100:00080000:3.0:1734017104.758549:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017104.758553:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017104.758554:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332738944:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017104.758557:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017104.758565:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332739328/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017104.758574:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017104.758577:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017104.758578:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017104.758582:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1367 00000100:00080000:3.0:1734017104.758584:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017104.758587:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017104.758591:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017104.758599:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332739456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017104.758607:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017104.758610:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017104.758612:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017104.758615:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1367 00000100:00080000:3.0:1734017104.758618:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017104.758619:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017104.758622:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017104.758627:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017104.758634:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332739584/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017104.758641:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017104/real 1734017104] req@ffff8800a73cf480 x1818247332738944/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017104.758643:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017104.758646:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017104.758648:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017104.758651:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1367 00000100:00080000:3.0:1734017104.758654:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017104.758657:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017104.758660:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332738944/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017104.758661:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017104.758669:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332739712/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017104.758671:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017104.758675:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017104.758680:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332738944:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017104.758687:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332739072:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017104.758704:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017104.758711:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017104/real 1734017104] req@ffff8800a73cca80 x1818247332739072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017104.758722:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332739072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017104.758731:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017104.758733:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017104.758737:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332739072:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017104.758743:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332739200:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017104.758756:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017104.758762:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017104/real 1734017104] req@ffff8800a73ced80 x1818247332739200/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017104.758772:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332739200/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017104.758779:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017104.758782:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017104.758785:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332739200:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017104.758810:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332739328:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017104.758822:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017104.758828:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017104/real 1734017104] req@ffff8800a73cea00 x1818247332739328/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017104.758839:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332739328/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017104.758846:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017104.758848:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017104.758851:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332739328:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017104.758857:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332739456:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017104.758869:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017104.758875:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017104/real 1734017104] req@ffff8800a73cd500 x1818247332739456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017104.758885:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332739456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017104.758892:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017104.758894:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017104.758897:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332739456:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017104.758902:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332739584:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017104.758914:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017104.758920:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017104/real 1734017104] req@ffff8800a73cfb80 x1818247332739584/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017104.758930:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332739584/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017104.758937:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017104.758939:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017104.758942:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332739584:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017104.758947:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332739712:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017104.758960:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017104.758965:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017104/real 1734017104] req@ffff8800a73cd880 x1818247332739712/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017104.758976:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332739712/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017104.758984:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017104.758986:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017104.758989:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332739712:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017104.916477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017105.918474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017106.920481:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017107.922490:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017108.924484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017109.766357:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017109.766366:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017109.766370:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017109.766381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1372 00000100:00080000:3.0:1734017109.766390:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017109.766398:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017109.766435:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332739840/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017109.766468:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017109.766475:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017109.766477:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017109.766482:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1372 00000100:00080000:3.0:1734017109.766485:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017109.766489:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017109.766500:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017109.766511:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332739968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017109.766521:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017109.766525:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017109.766526:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017109.766530:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1372 00000100:00080000:3.0:1734017109.766532:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017109.766536:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017109.766541:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017109.766549:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332740096/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017109.766558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017109.766561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017109.766563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017109.766566:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1372 00000100:00080000:3.0:1734017109.766569:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017109.766572:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332739840:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017109.766572:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017109.766577:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017109.766585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332740224/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017109.766593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017109.766596:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017109.766598:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017109.766601:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1372 00000100:00080000:3.0:1734017109.766604:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017109.766607:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017109.766611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017109.766619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332740352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017109.766627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017109.766630:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017109.766631:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017109.766635:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1372 00000100:00080000:3.0:1734017109.766638:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017109.766640:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017109.766641:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017109.766645:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017109.766653:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332740480/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017109.766661:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017109.766662:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017109/real 1734017109] req@ffff8800a73cf800 x1818247332739840/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017109.766664:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017109.766667:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017109.766671:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1372 00000100:00080000:3.0:1734017109.766674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017109.766677:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017109.766681:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017109.766682:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332739840/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017109.766690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332740608/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017109.766693:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017109.766696:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017109.766701:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332739840:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017109.766708:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332739968:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017109.766726:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017109.766732:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017109/real 1734017109] req@ffff8800a73cdf80 x1818247332739968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017109.766743:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332739968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017109.766752:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017109.766755:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017109.766758:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332739968:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017109.766782:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332740096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017109.766796:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017109.766802:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017109/real 1734017109] req@ffff8800a73cc700 x1818247332740096/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017109.766813:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332740096/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017109.766821:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017109.766823:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017109.766826:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332740096:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017109.766832:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332740224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017109.766843:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017109.766849:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017109/real 1734017109] req@ffff8800a73ce680 x1818247332740224/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017109.766858:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332740224/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017109.766866:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017109.766868:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017109.766871:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332740224:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017109.766877:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332740352:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017109.766889:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017109.766896:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017109/real 1734017109] req@ffff8800a73cdc00 x1818247332740352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017109.766906:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332740352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017109.766914:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017109.766916:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017109.766919:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332740352:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017109.766924:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332740480:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017109.766935:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017109.766941:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017109/real 1734017109] req@ffff8800a73cf100 x1818247332740480/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017109.766951:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332740480/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017109.766958:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017109.766960:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017109.766963:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332740480:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017109.766968:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332740608:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017109.766979:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017109.766985:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017109/real 1734017109] req@ffff8800a73cce00 x1818247332740608/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017109.766995:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332740608/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017109.767003:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017109.767005:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017109.767008:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332740608:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017109.926480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017110.928484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017111.930505:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017112.932507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017113.934495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017114.774340:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017114.774348:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017114.774353:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017114.774363:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1377 00000100:00080000:3.0:1734017114.774373:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017114.774381:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017114.774417:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332740736/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017114.774450:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017114.774455:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017114.774457:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017114.774461:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1377 00000100:00080000:3.0:1734017114.774465:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017114.774469:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017114.774478:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017114.774490:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332740864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017114.774501:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017114.774504:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017114.774506:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017114.774509:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1377 00000100:00080000:3.0:1734017114.774512:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017114.774515:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017114.774520:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017114.774528:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332740992/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017114.774536:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017114.774539:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017114.774541:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017114.774544:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1377 00000100:00080000:3.0:1734017114.774547:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017114.774550:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017114.774554:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017114.774557:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332740736:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017114.774562:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332741120/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017114.774571:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017114.774575:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017114.774577:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017114.774580:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1377 00000100:00080000:3.0:1734017114.774583:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017114.774587:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017114.774591:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017114.774599:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332741248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017114.774607:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017114.774610:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017114.774611:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017114.774615:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1377 00000100:00080000:3.0:1734017114.774618:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017114.774621:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734017114.774622:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017114.774626:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017114.774633:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332741376/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017114.774641:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017114.774644:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734017114.774645:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017114/real 1734017114] req@ffff8800a73cc380 x1818247332740736/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017114.774645:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017114.774649:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1377 00000100:00080000:3.0:1734017114.774651:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017114.774654:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017114.774658:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017114.774665:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332740736/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017114.774666:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332741504/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017114.774676:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017114.774679:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017114.774684:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332740736:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017114.774691:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332740864:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017114.774708:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017114.774715:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017114/real 1734017114] req@ffff8800a73cce00 x1818247332740864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017114.774726:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332740864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017114.774735:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017114.774738:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017114.774741:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332740864:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017114.774747:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332740992:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017114.774760:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017114.774766:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017114/real 1734017114] req@ffff8800a73cf100 x1818247332740992/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017114.774776:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332740992/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017114.774784:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017114.774786:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017114.774789:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332740992:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017114.774814:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332741120:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017114.774826:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017114.774832:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017114/real 1734017114] req@ffff8800a73cdc00 x1818247332741120/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017114.774842:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332741120/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017114.774850:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017114.774852:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017114.774855:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332741120:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017114.774860:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332741248:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017114.774873:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017114.774879:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017114/real 1734017114] req@ffff8800a73ce680 x1818247332741248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017114.774889:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332741248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017114.774897:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017114.774899:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017114.774902:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332741248:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017114.774907:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332741376:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017114.774918:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017114.774924:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017114/real 1734017114] req@ffff8800a73cc700 x1818247332741376/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017114.774933:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332741376/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017114.774940:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017114.774942:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017114.774945:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332741376:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017114.774950:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332741504:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017114.774963:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017114.774969:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017114/real 1734017114] req@ffff8800a73cdf80 x1818247332741504/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017114.774979:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332741504/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017114.774987:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017114.774989:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017114.774992:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332741504:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017114.936498:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017115.938433:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017116.940465:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017117.942451:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017118.944514:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017119.782336:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017119.782344:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017119.782349:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017119.782360:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1382 00000100:00080000:3.0:1734017119.782369:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017119.782377:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017119.782414:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332741632/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017119.782448:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017119.782454:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017119.782457:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017119.782464:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1382 00000100:00080000:3.0:1734017119.782469:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017119.782475:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017119.782487:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017119.782505:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332741760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017119.782510:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332741632:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017119.782521:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017119.782526:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017119.782528:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017119.782533:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1382 00000100:00080000:3.0:1734017119.782538:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017119.782543:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017119.782549:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017119.782560:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332741888/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017119.782572:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017119.782576:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017119.782579:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734017119.782582:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017119.782584:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1382 00000100:00080000:3.0:1734017119.782588:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017119.782592:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017119.782598:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017119.782606:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017119/real 1734017119] req@ffff8800a73cf800 x1818247332741632/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017119.782610:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332742016/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017119.782624:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017119.782628:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1734017119.782631:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332741632/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017119.782631:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017119.782636:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1382 00000100:00080000:3.0:1734017119.782640:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1734017119.782645:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017119.782645:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:1.0:1734017119.782650:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000008:00100000:3.0:1734017119.782652:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734017119.782656:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332741632:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017119.782663:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332742144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017119.782683:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332741760:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017119.782685:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017119.782690:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017119.782692:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017119.782698:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1382 00000100:00080000:3.0:1734017119.782702:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017119.782708:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017119.782708:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017119.782714:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017119.782719:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017119/real 1734017119] req@ffff8800a73cd880 x1818247332741760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017119.782727:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332742272/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017119.782737:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332741760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017119.782741:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017119.782746:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017119.782748:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734017119.782751:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017119.782754:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1382 00000100:00080000:1.0:1734017119.782755:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734017119.782758:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017119.782761:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332741760:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017119.782763:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017119.782769:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332741888:192.168.201.152@tcp:8:kworker.0 00000008:00100000:3.0:1734017119.782769:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017119.782782:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332742400/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017119.782791:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017119.782801:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017119/real 1734017119] req@ffff8800a73cfb80 x1818247332741888/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017119.782818:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332741888/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017119.782831:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017119.782834:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017119.782840:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332741888:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017119.782849:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332742016:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017119.782877:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017119.782887:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017119/real 1734017119] req@ffff8800a73cd500 x1818247332742016/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017119.782898:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332742016/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017119.782905:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017119.782908:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017119.782911:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332742016:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017119.782928:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332742144:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017119.782943:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017119.782949:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017119/real 1734017119] req@ffff8800a73cea00 x1818247332742144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017119.782960:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332742144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017119.782968:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017119.782970:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017119.782973:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332742144:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017119.783022:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332742272:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017119.783039:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017119.783046:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017119/real 1734017119] req@ffff8800a73ced80 x1818247332742272/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017119.783056:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332742272/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017119.783064:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017119.783066:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017119.783069:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332742272:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017119.783074:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332742400:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017119.783085:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017119.783091:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017119/real 1734017119] req@ffff8800a73cca80 x1818247332742400/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017119.783102:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332742400/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017119.783110:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017119.783112:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017119.783115:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332742400:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017119.946440:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017120.948503:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017121.950459:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017122.952519:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017123.954501:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017124.790358:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017124.790367:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017124.790372:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017124.790382:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1387 00000100:00080000:3.0:1734017124.790397:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017124.790405:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017124.790440:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332742528/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017124.790473:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017124.790477:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017124.790479:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017124.790484:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1387 00000100:00080000:3.0:1734017124.790487:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017124.790491:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017124.790500:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017124.790512:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332742656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017124.790522:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017124.790525:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017124.790527:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017124.790531:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1387 00000100:00080000:3.0:1734017124.790534:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017124.790537:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017124.790542:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017124.790550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332742784/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017124.790558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017124.790562:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017124.790564:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017124.790568:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1387 00000100:00080000:3.0:1734017124.790570:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017124.790574:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017124.790579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017124.790583:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332742528:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017124.790587:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332742912/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017124.790595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017124.790598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017124.790600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017124.790603:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1387 00000100:00080000:3.0:1734017124.790605:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017124.790609:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017124.790613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017124.790621:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332743040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017124.790629:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017124.790632:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017124.790634:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017124.790638:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1387 00000100:00080000:3.0:1734017124.790640:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017124.790643:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017124.790648:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017124.790649:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017124.790656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332743168/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017124.790664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017124.790667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017124.790669:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017124.790671:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017124/real 1734017124] req@ffff8800a73cf480 x1818247332742528/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017124.790672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1387 00000100:00080000:3.0:1734017124.790675:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017124.790678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017124.790682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017124.790689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332743296/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017124.790691:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332742528/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017124.790702:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017124.790705:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017124.790710:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332742528:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017124.790717:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332742656:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017124.790734:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017124.790741:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017124/real 1734017124] req@ffff8800a73cca80 x1818247332742656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017124.790752:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332742656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017124.790761:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017124.790764:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017124.790767:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332742656:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017124.790773:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332742784:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017124.790787:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017124.790793:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017124/real 1734017124] req@ffff8800a73ced80 x1818247332742784/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017124.790803:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332742784/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017124.790811:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017124.790813:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017124.790816:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332742784:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017124.790841:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332742912:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017124.790854:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017124.790860:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017124/real 1734017124] req@ffff8800a73cea00 x1818247332742912/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017124.790871:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332742912/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017124.790878:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017124.790880:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017124.790883:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332742912:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017124.790930:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332743040:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017124.790949:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017124.790957:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017124/real 1734017124] req@ffff8800a73cd500 x1818247332743040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017124.790968:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332743040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017124.790976:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017124.790978:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017124.790981:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332743040:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017124.790986:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332743168:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017124.790998:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017124.791004:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017124/real 1734017124] req@ffff8800a73cfb80 x1818247332743168/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017124.791014:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332743168/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017124.791021:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017124.791024:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017124.791027:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332743168:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017124.791032:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332743296:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017124.791044:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017124.791050:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017124/real 1734017124] req@ffff8800a73cd880 x1818247332743296/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017124.791061:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332743296/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017124.791069:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017124.791071:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017124.791074:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332743296:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017124.956470:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017125.958440:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017126.960501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017127.962482:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017128.964488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017129.798368:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017129.798377:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017129.798382:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017129.798392:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1392 00000100:00080000:3.0:1734017129.798402:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017129.798409:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017129.798446:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332743424/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017129.798479:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017129.798487:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017129.798489:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017129.798493:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1392 00000100:00080000:3.0:1734017129.798497:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017129.798501:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017129.798510:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017129.798522:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332743552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017129.798531:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017129.798535:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017129.798536:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017129.798540:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1392 00000100:00080000:3.0:1734017129.798543:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017129.798546:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017129.798551:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017129.798559:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332743680/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017129.798567:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017129.798570:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017129.798572:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017129.798575:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1392 00000100:00080000:3.0:1734017129.798578:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017129.798581:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017129.798584:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332743424:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017129.798587:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017129.798594:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332743808/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017129.798604:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017129.798607:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017129.798608:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017129.798612:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1392 00000100:00080000:3.0:1734017129.798614:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017129.798618:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017129.798622:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017129.798629:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332743936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017129.798637:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017129.798641:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017129.798642:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017129.798646:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1392 00000800:00000100:1.0:1734017129.798648:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017129.798649:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017129.798652:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017129.798656:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017129.798664:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332744064/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017129.798671:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017129/real 1734017129] req@ffff8800a73cf800 x1818247332743424/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017129.798672:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017129.798676:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017129.798678:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017129.798682:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1392 00000100:00080000:3.0:1734017129.798685:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017129.798688:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017129.798691:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332743424/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017129.798692:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017129.798700:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332744192/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017129.798702:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017129.798705:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017129.798710:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332743424:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017129.798718:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332743552:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017129.798734:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017129.798741:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017129/real 1734017129] req@ffff8800a73cdf80 x1818247332743552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017129.798751:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332743552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017129.798760:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017129.798763:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017129.798766:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332743552:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017129.798772:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332743680:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017129.798785:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017129.798791:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017129/real 1734017129] req@ffff8800a73cc700 x1818247332743680/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017129.798800:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332743680/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017129.798850:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017129.798852:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017129.798856:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332743680:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017129.798881:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332743808:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017129.798898:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017129.798906:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017129/real 1734017129] req@ffff8800a73ce680 x1818247332743808/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017129.798916:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332743808/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017129.798923:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017129.798926:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017129.798929:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332743808:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017129.798934:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332743936:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017129.798946:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017129.798953:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017129/real 1734017129] req@ffff8800a73cdc00 x1818247332743936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017129.798963:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332743936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017129.798971:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017129.798973:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017129.798976:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332743936:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017129.798981:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332744064:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017129.798992:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017129.798998:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017129/real 1734017129] req@ffff8800a73cf100 x1818247332744064/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017129.799008:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332744064/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017129.799014:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017129.799016:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017129.799019:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332744064:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017129.799024:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332744192:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017129.799036:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017129.799042:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017129/real 1734017129] req@ffff8800a73cce00 x1818247332744192/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017129.799053:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332744192/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017129.799061:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017129.799063:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017129.799066:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332744192:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017129.966469:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017130.968481:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017131.970497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017132.972479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017133.974388:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017134.806345:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017134.806354:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017134.806358:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017134.806368:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1397 00000100:00080000:3.0:1734017134.806378:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017134.806386:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017134.806421:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332744320/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017134.806453:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017134.806458:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017134.806460:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017134.806464:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1397 00000100:00080000:3.0:1734017134.806468:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017134.806472:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017134.806481:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017134.806493:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332744448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017134.806504:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017134.806507:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017134.806509:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017134.806512:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1397 00000100:00080000:3.0:1734017134.806514:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017134.806518:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017134.806523:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017134.806530:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332744576/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017134.806539:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017134.806542:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017134.806543:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017134.806546:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1397 00000100:00080000:3.0:1734017134.806549:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017134.806552:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017134.806556:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017134.806561:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332744320:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017134.806564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332744704/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017134.806573:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017134.806576:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017134.806578:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017134.806581:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1397 00000100:00080000:3.0:1734017134.806584:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017134.806588:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017134.806592:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017134.806600:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332744832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017134.806608:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017134.806611:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017134.806612:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017134.806616:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1397 00000100:00080000:3.0:1734017134.806619:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017134.806622:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017134.806626:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017134.806627:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017134.806633:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332744960/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017134.806641:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017134.806644:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017134.806646:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017134.806649:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017134/real 1734017134] req@ffff8800a73cc380 x1818247332744320/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017134.806649:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1397 00000100:00080000:3.0:1734017134.806652:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017134.806655:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017134.806659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017134.806667:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332745088/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017134.806669:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332744320/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017134.806680:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017134.806683:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017134.806688:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332744320:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017134.806695:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332744448:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017134.806712:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017134.806719:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017134/real 1734017134] req@ffff8800a73cce00 x1818247332744448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017134.806760:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332744448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017134.806769:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017134.806772:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017134.806775:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332744448:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017134.806782:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332744576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017134.806797:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017134.806804:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017134/real 1734017134] req@ffff8800a73cf100 x1818247332744576/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017134.806814:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332744576/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017134.806822:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017134.806824:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017134.806827:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332744576:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017134.806851:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332744704:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017134.806864:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017134.806870:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017134/real 1734017134] req@ffff8800a73cdc00 x1818247332744704/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017134.806880:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332744704/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017134.806888:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017134.806890:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017134.806893:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332744704:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017134.806900:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332744832:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017134.806912:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017134.806918:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017134/real 1734017134] req@ffff8800a73ce680 x1818247332744832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017134.806929:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332744832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017134.806936:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017134.806938:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017134.806941:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332744832:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017134.806947:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332744960:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017134.806958:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017134.806964:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017134/real 1734017134] req@ffff8800a73cc700 x1818247332744960/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017134.806974:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332744960/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017134.806981:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017134.806983:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017134.806986:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332744960:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017134.806991:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332745088:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017134.807003:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017134.807009:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017134/real 1734017134] req@ffff8800a73cdf80 x1818247332745088/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017134.807020:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332745088/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017134.807027:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017134.807030:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017134.807032:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332745088:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017134.976471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017135.978455:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017136.980513:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017137.982444:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017138.984469:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017139.814374:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017139.814383:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017139.814388:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017139.814398:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1402 00000100:00080000:3.0:1734017139.814407:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017139.814415:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017139.814450:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332745216/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017139.814482:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017139.814486:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017139.814488:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017139.814493:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1402 00000100:00080000:3.0:1734017139.814496:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017139.814500:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017139.814509:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017139.814520:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332745344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017139.814530:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017139.814533:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017139.814535:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017139.814539:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1402 00000100:00080000:3.0:1734017139.814543:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017139.814546:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017139.814552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017139.814559:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332745472/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017139.814567:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017139.814570:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017139.814572:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017139.814575:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1402 00000100:00080000:3.0:1734017139.814577:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017139.814581:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017139.814585:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017139.814587:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332745216:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017139.814593:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332745600/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017139.814602:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017139.814605:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017139.814606:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017139.814610:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1402 00000100:00080000:3.0:1734017139.814613:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017139.814616:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017139.814620:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017139.814627:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332745728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017139.814676:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017139.814679:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017139.814681:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017139.814685:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1402 00000100:00080000:3.0:1734017139.814688:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017139.814692:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734017139.814694:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017139.814697:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017139.814706:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332745856/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017139.814714:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017139.814717:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017139/real 1734017139] req@ffff8800a73cf800 x1818247332745216/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017139.814717:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017139.814719:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017139.814722:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1402 00000100:00080000:3.0:1734017139.814725:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017139.814728:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017139.814732:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017139.814738:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332745216/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017139.814741:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332745984/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017139.814749:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017139.814752:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017139.814757:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332745216:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017139.814765:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332745344:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017139.814784:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017139.814791:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017139/real 1734017139] req@ffff8800a73cd880 x1818247332745344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017139.814802:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332745344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017139.814811:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017139.814814:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017139.814817:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332745344:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017139.814822:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332745472:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017139.814836:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017139.814842:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017139/real 1734017139] req@ffff8800a73cfb80 x1818247332745472/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017139.814852:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332745472/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017139.814859:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017139.814862:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017139.814865:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332745472:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017139.814890:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332745600:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017139.814902:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017139.814909:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017139/real 1734017139] req@ffff8800a73cd500 x1818247332745600/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017139.814919:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332745600/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017139.814926:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017139.814928:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017139.814931:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332745600:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017139.814937:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332745728:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017139.814949:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017139.814955:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017139/real 1734017139] req@ffff8800a73cea00 x1818247332745728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017139.814966:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332745728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017139.814974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017139.814976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017139.814979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332745728:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017139.814983:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332745856:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017139.814995:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017139.815000:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017139/real 1734017139] req@ffff8800a73ced80 x1818247332745856/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017139.815010:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332745856/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017139.815017:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017139.815019:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017139.815022:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332745856:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017139.815027:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332745984:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017139.815038:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017139.815044:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017139/real 1734017139] req@ffff8800a73cca80 x1818247332745984/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017139.815055:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332745984/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017139.815063:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017139.815065:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017139.815068:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332745984:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017139.986473:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017140.988518:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017141.990495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017142.992504:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017143.994485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017144.822371:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017144.822380:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017144.822385:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017144.822395:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1407 00000100:00080000:3.0:1734017144.822405:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017144.822413:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017144.822449:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332746112/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017144.822483:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017144.822487:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017144.822489:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017144.822494:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1407 00000100:00080000:3.0:1734017144.822497:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017144.822501:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017144.822511:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017144.822524:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332746240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017144.822534:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017144.822537:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017144.822539:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017144.822543:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1407 00000100:00080000:3.0:1734017144.822546:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017144.822582:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017144.822588:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017144.822598:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332746368/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017144.822608:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017144.822612:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017144.822613:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017144.822617:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1407 00000100:00100000:1.0:1734017144.822619:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332746112:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017144.822620:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017144.822623:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017144.822628:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017144.822635:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332746496/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017144.822644:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017144.822647:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017144.822648:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017144.822652:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1407 00000100:00080000:3.0:1734017144.822655:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017144.822658:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017144.822662:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017144.822670:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332746624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017144.822678:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017144.822681:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017144.822683:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017144.822686:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1407 00000100:00080000:3.0:1734017144.822689:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017144.822692:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017144.822698:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017144.822705:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332746752/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017144.822711:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332746240:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017144.822713:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017144.822716:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017144.822718:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017144.822721:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1407 00000100:00080000:3.0:1734017144.822724:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017144.822727:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017144.822731:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017144.822739:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332746880/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017144.822740:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332746368:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017144.822756:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332746496:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017144.822770:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332746624:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017144.822785:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332746752:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017144.822800:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332746880:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017144.996499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017145.998442:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017147.000443:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734017147.828485:0:1956:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734017147.828575:0:1956:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734017147.828596:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017147.828630:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017147.828638:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017147.828646:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017147.828653:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017147.828659:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017147.828664:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734017147.828733:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017144/real 1734017147] req@ffff8800a73cd880 x1818247332746880/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017147.828762:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332746880/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017147.828782:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017147.828786:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017147.828791:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332746880:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017147.828799:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017144/real 1734017147] req@ffff8800a73ced80 x1818247332746368/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017147.828810:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332746368/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017147.828818:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017147.828821:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017147.828825:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332746368:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017147.828831:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017144/real 1734017147] req@ffff8800a73cea00 x1818247332746496/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017147.828841:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332746496/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017147.828858:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017147.828861:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017147.828864:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332746496:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017147.828870:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017144/real 1734017147] req@ffff8800a73cd500 x1818247332746624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017147.828881:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332746624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017147.828897:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017147.828901:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017147.828906:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332746624:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017147.828916:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017144/real 1734017147] req@ffff8800a73cfb80 x1818247332746752/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017147.828933:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332746752/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017147.828945:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017147.828948:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017147.828953:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332746752:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017147.828963:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017144/real 1734017147] req@ffff8800a73cf480 x1818247332746112/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017147.828981:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332746112/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017147.828993:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017147.828996:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017147.829001:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332746112:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017147.829011:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017144/real 1734017147] req@ffff8800a73cca80 x1818247332746240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017147.829028:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332746240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017147.829050:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017147.829054:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017147.829060:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332746240:192.168.201.152@tcp:38:kworker.0 00010000:00010000:1.0:1734017148.002485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017149.004454:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017149.830372:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017149.830381:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017149.830386:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017149.830395:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1412 00000100:00080000:3.0:1734017149.830404:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017149.830412:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017149.830447:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332747008/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017149.830507:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017149.830514:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017149.830517:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017149.830521:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1412 00000100:00080000:3.0:1734017149.830525:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017149.830529:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017149.830539:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017149.830551:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332747136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017149.830561:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017149.830565:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017149.830566:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017149.830570:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1412 00000100:00080000:3.0:1734017149.830572:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017149.830575:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017149.830580:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017149.830588:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332747264/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017149.830597:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017149.830600:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017149.830601:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017149.830605:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1412 00000100:00080000:3.0:1734017149.830607:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017149.830610:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017149.830615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017149.830622:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332747008:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017149.830623:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332747392/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017149.830632:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017149.830635:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017149.830637:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017149.830640:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1412 00000100:00080000:3.0:1734017149.830643:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017149.830646:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017149.830650:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017149.830658:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332747520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017149.830666:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017149.830669:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017149.830670:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017149.830674:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1412 00000100:00080000:3.0:1734017149.830679:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017149.830682:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017149.830686:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017149.830690:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017149.830694:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332747648/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017149.830701:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017149.830705:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017149.830707:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017149.830711:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1412 00000100:00100000:1.0:1734017149.830713:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017149/real 1734017149] req@ffff8800a73cf800 x1818247332747008/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017149.830714:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017149.830717:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017149.830721:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017149.830730:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332747776/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017149.830733:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332747008/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017149.830743:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017149.830746:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017149.830751:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332747008:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017149.830759:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332747136:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017149.830775:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017149.830782:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017149/real 1734017149] req@ffff8800a73cdf80 x1818247332747136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017149.830793:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332747136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017149.830801:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017149.830803:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017149.830808:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332747136:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017149.830814:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332747264:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017149.830828:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017149.830833:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017149/real 1734017149] req@ffff8800a73cc700 x1818247332747264/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017149.830844:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332747264/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017149.830851:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017149.830854:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017149.830857:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332747264:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017149.830880:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332747392:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017149.830892:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017149.830898:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017149/real 1734017149] req@ffff8800a73ce680 x1818247332747392/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017149.830908:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332747392/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017149.830916:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017149.830918:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017149.830921:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332747392:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017149.830926:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332747520:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017149.830937:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017149.830943:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017149/real 1734017149] req@ffff8800a73cdc00 x1818247332747520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017149.830954:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332747520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017149.830961:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017149.830964:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017149.830966:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332747520:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017149.830971:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332747648:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017149.830982:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017149.830988:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017149/real 1734017149] req@ffff8800a73cf100 x1818247332747648/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017149.830998:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332747648/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017149.831005:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017149.831007:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017149.831010:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332747648:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017149.831014:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332747776:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017149.831026:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017149.831032:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017149/real 1734017149] req@ffff8800a73cce00 x1818247332747776/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017149.831042:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332747776/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017149.831057:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017149.831059:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017149.831062:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332747776:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017150.006506:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017151.008478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017152.010530:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017153.012506:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017154.014491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017154.838405:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017154.838414:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017154.838418:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017154.838429:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1417 00000100:00080000:3.0:1734017154.838441:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017154.838449:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017154.838484:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332747904/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017154.838519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017154.838524:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017154.838526:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017154.838530:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1417 00000100:00080000:3.0:1734017154.838534:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017154.838538:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017154.838547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017154.838560:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332748032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017154.838572:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017154.838575:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017154.838577:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017154.838580:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1417 00000100:00080000:3.0:1734017154.838583:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017154.838586:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017154.838591:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017154.838599:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332748160/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017154.838608:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017154.838611:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017154.838612:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017154.838616:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1417 00000100:00080000:3.0:1734017154.838618:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017154.838621:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017154.838626:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017154.838633:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332748288/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017154.838642:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017154.838643:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332747904:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017154.838645:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017154.838646:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017154.838650:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1417 00000100:00080000:3.0:1734017154.838653:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017154.838656:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017154.838661:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017154.838668:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332748416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017154.838676:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017154.838679:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017154.838681:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017154.838685:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1417 00000100:00080000:3.0:1734017154.838687:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017154.838691:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017154.838695:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017154.838702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332748544/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017154.838710:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017154.838710:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017154.838713:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017154.838714:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017154.838718:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1417 00000100:00080000:3.0:1734017154.838720:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017154.838723:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017154.838728:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017154.838733:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017154/real 1734017154] req@ffff8800a73cc380 x1818247332747904/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017154.838735:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332748672/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017154.838753:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332747904/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017154.838764:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017154.838767:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017154.838772:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332747904:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017154.838779:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332748032:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017154.838796:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017154.838803:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017154/real 1734017154] req@ffff8800a73cce00 x1818247332748032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017154.838814:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332748032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017154.838822:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017154.838825:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017154.838829:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332748032:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017154.838835:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332748160:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017154.838849:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017154.838855:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017154/real 1734017154] req@ffff8800a73cf100 x1818247332748160/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017154.838865:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332748160/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017154.838873:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017154.838875:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017154.838879:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332748160:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017154.838903:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332748288:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017154.838915:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017154.838921:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017154/real 1734017154] req@ffff8800a73cdc00 x1818247332748288/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017154.838932:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332748288/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017154.838939:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017154.838941:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017154.838944:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332748288:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017154.838949:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332748416:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017154.838964:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017154.838970:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017154/real 1734017154] req@ffff8800a73ce680 x1818247332748416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017154.838980:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332748416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017154.838988:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017154.838990:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017154.838993:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332748416:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017154.838997:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332748544:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017154.839009:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017154.839015:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017154/real 1734017154] req@ffff8800a73cc700 x1818247332748544/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017154.839025:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332748544/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017154.839032:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017154.839034:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017154.839037:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332748544:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017154.839042:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332748672:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017154.839054:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017154.839060:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017154/real 1734017154] req@ffff8800a73cdf80 x1818247332748672/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017154.839076:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332748672/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017154.839084:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017154.839086:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017154.839089:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332748672:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017155.016514:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017156.018518:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017157.020627:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017158.022544:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017159.024540:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017159.846389:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017159.846397:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017159.846402:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017159.846413:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1422 00000100:00080000:3.0:1734017159.846424:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017159.846433:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017159.846469:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332748800/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017159.846504:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017159.846508:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017159.846510:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017159.846515:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1422 00000100:00080000:3.0:1734017159.846518:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017159.846522:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017159.846532:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017159.846542:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332748928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017159.846553:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017159.846556:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017159.846558:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017159.846561:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1422 00000100:00080000:3.0:1734017159.846565:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017159.846569:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017159.846573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017159.846582:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332749056/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017159.846590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017159.846593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017159.846595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017159.846598:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1422 00000100:00080000:3.0:1734017159.846601:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017159.846604:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017159.846608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017159.846615:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332749184/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017159.846624:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017159.846627:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017159.846629:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017159.846631:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332748800:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017159.846632:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1422 00000100:00080000:3.0:1734017159.846635:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017159.846638:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017159.846643:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017159.846650:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332749312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017159.846665:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017159.846668:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017159.846670:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017159.846674:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1422 00000100:00080000:3.0:1734017159.846677:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017159.846680:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017159.846684:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017159.846691:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332749440/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017159.846698:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017159.846699:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017159.846702:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017159.846704:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017159.846707:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1422 00000100:00080000:3.0:1734017159.846710:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017159.846713:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017159.846717:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017159.846720:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017159/real 1734017159] req@ffff8800a73cf800 x1818247332748800/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017159.846726:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332749568/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017159.846739:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332748800/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017159.846750:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017159.846753:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017159.846758:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332748800:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017159.846765:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332748928:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017159.846782:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017159.846788:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017159/real 1734017159] req@ffff8800a73cca80 x1818247332748928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017159.846799:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332748928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017159.846808:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017159.846810:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017159.846814:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332748928:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017159.846820:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332749056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017159.846833:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017159.846839:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017159/real 1734017159] req@ffff8800a73cf480 x1818247332749056/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017159.846849:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332749056/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017159.846856:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017159.846859:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017159.846863:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332749056:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017159.846887:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332749184:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017159.846899:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017159.846906:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017159/real 1734017159] req@ffff8800a73cfb80 x1818247332749184/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017159.846916:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332749184/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017159.846924:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017159.846926:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017159.846929:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332749184:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017159.846934:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332749312:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017159.846946:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017159.846953:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017159/real 1734017159] req@ffff8800a73cd500 x1818247332749312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017159.846963:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332749312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017159.846971:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017159.846973:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017159.846976:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332749312:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017159.846980:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332749440:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017159.846992:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017159.846998:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017159/real 1734017159] req@ffff8800a73cea00 x1818247332749440/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017159.847007:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332749440/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017159.847014:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017159.847016:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017159.847019:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332749440:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017159.847024:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332749568:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017159.847036:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017159.847042:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017159/real 1734017159] req@ffff8800a73ced80 x1818247332749568/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017159.847052:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332749568/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017159.847060:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017159.847062:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017159.847065:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332749568:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017160.026520:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017161.028492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017162.030525:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017163.032538:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017164.034476:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017164.854361:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017164.854370:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017164.854374:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017164.854385:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1427 00000100:00080000:3.0:1734017164.854396:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017164.854404:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017164.854440:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332749696/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017164.854474:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017164.854479:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017164.854481:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017164.854485:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1427 00000100:00080000:3.0:1734017164.854488:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017164.854492:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017164.854503:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017164.854516:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332749824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017164.854526:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017164.854529:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017164.854531:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017164.854535:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1427 00000100:00080000:3.0:1734017164.854538:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017164.854541:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017164.854546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017164.854554:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332749952/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017164.854562:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017164.854566:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017164.854568:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017164.854571:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1427 00000100:00080000:3.0:1734017164.854574:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017164.854577:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017164.854581:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017164.854585:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332749696:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017164.854589:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332750080/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017164.854598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017164.854601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017164.854602:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017164.854606:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1427 00000100:00080000:3.0:1734017164.854608:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017164.854611:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017164.854616:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017164.854623:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332750208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017164.854631:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017164.854634:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017164.854636:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017164.854639:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1427 00000100:00080000:3.0:1734017164.854642:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017164.854645:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017164.854650:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017164.854651:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017164.854658:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332750336/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017164.854666:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017164.854669:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017164.854671:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017164.854674:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017164/real 1734017164] req@ffff8800a73cd880 x1818247332749696/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017164.854674:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1427 00000100:00080000:3.0:1734017164.854677:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017164.854680:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017164.854684:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017164.854692:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332750464/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017164.854695:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332749696/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017164.854705:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017164.854708:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017164.854713:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332749696:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017164.854721:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332749824:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017164.854738:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017164.854745:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017164/real 1734017164] req@ffff8800a73ced80 x1818247332749824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017164.854757:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332749824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017164.854771:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017164.854778:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017164.854782:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332749824:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017164.854788:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332749952:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017164.854802:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017164.854807:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017164/real 1734017164] req@ffff8800a73cea00 x1818247332749952/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017164.854817:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332749952/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017164.854825:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017164.854827:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017164.854831:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332749952:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017164.854858:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332750080:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017164.854870:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017164.854877:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017164/real 1734017164] req@ffff8800a73cd500 x1818247332750080/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017164.854888:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332750080/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017164.854895:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017164.854897:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017164.854900:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332750080:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017164.854907:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332750208:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017164.854919:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017164.854924:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017164/real 1734017164] req@ffff8800a73cfb80 x1818247332750208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017164.854935:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332750208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017164.854942:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017164.854945:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017164.854948:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332750208:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017164.854953:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332750336:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017164.854964:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017164.854970:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017164/real 1734017164] req@ffff8800a73cf480 x1818247332750336/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017164.854980:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332750336/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017164.854987:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017164.854989:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017164.854992:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332750336:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017164.854997:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332750464:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017164.855009:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017164.855015:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017164/real 1734017164] req@ffff8800a73cca80 x1818247332750464/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017164.855026:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332750464/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017164.855034:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017164.855036:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017164.855039:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332750464:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017165.036491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017166.038481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017167.040472:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017168.042497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017169.044475:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017169.862352:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017169.862361:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017169.862366:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017169.862375:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1432 00000100:00080000:3.0:1734017169.862385:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017169.862392:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017169.862430:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332750592/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017169.862465:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017169.862471:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017169.862473:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017169.862478:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1432 00000100:00080000:3.0:1734017169.862481:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017169.862485:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017169.862496:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017169.862513:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332750720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017169.862528:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017169.862533:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017169.862536:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017169.862541:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1432 00000100:00080000:3.0:1734017169.862545:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017169.862551:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017169.862558:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017169.862572:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332750848/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017169.862576:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332750592:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017169.862587:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017169.862591:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017169.862594:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017169.862599:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1432 00000100:00080000:3.0:1734017169.862603:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017169.862609:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017169.862617:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017169.862632:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332750976/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017169.862643:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017169.862646:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017169.862651:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017169.862654:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017169.862659:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1432 00000100:00080000:3.0:1734017169.862664:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017169.862665:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017169/real 1734017169] req@ffff8800a73cf800 x1818247332750592/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017169.862669:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017169.862675:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734017169.862685:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332750592/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017169.862688:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332751104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017169.862696:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017169.862699:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734017169.862700:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017169.862704:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332750592:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017169.862704:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017169.862707:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017169.862711:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332750720:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017169.862713:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1432 00000100:00080000:3.0:1734017169.862716:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017169.862722:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017169.862728:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017169.862729:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017169.862735:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017169/real 1734017169] req@ffff8800a73cdf80 x1818247332750720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017169.862740:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332751232/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017169.862747:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332750720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017169.862752:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1734017169.862755:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017169.862757:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1734017169.862758:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017169.862761:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332750720:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017169.862761:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017169.862766:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1432 00000100:00080000:3.0:1734017169.862770:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017169.862775:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017169.862782:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017169.862784:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332750848:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734017169.862797:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332751360/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017169.862798:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017169.862805:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017169/real 1734017169] req@ffff8800a73cc700 x1818247332750848/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017169.862816:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332750848/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017169.862824:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017169.862826:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017169.862829:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332750848:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017169.862834:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332750976:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017169.862846:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017169.862851:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017169/real 1734017169] req@ffff8800a73ce680 x1818247332750976/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017169.862861:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332750976/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017169.862868:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017169.862870:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017169.862873:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332750976:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017169.862880:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332751104:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017169.862893:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017169.862899:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017169/real 1734017169] req@ffff8800a73cdc00 x1818247332751104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017169.862910:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332751104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017169.862918:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017169.862921:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017169.862923:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332751104:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017169.862929:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332751232:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017169.862940:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017169.862946:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017169/real 1734017169] req@ffff8800a73cf100 x1818247332751232/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017169.862956:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332751232/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017169.862963:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017169.862965:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017169.862968:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332751232:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017169.862982:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332751360:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017169.862994:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017169.863000:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017169/real 1734017169] req@ffff8800a73cce00 x1818247332751360/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017169.863012:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332751360/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017169.863020:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017169.863022:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017169.863025:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332751360:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017170.046496:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017171.048401:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017172.050442:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017173.052474:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017174.054465:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017174.870361:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017174.870370:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017174.870374:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017174.870385:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1437 00000100:00080000:3.0:1734017174.870394:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017174.870402:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017174.870437:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332751488/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017174.870472:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017174.870476:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017174.870479:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017174.870483:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1437 00000100:00080000:3.0:1734017174.870486:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017174.870490:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017174.870500:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017174.870512:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332751616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017174.870523:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017174.870527:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017174.870528:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017174.870532:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1437 00000100:00080000:3.0:1734017174.870534:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017174.870537:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017174.870542:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017174.870550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332751744/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017174.870559:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017174.870562:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017174.870563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017174.870567:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1437 00000100:00080000:3.0:1734017174.870569:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017174.870572:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017174.870576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017174.870584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332751872/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017174.870590:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332751488:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017174.870592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017174.870595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017174.870597:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017174.870600:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1437 00000100:00080000:3.0:1734017174.870604:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017174.870607:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017174.870611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017174.870619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332752000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017174.870627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017174.870630:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017174.870631:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017174.870635:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1437 00000100:00080000:3.0:1734017174.870638:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017174.870641:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017174.870645:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017174.870652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332752128/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017174.870657:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017174.870660:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017174.870663:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017174.870664:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017174.870668:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1437 00000100:00080000:3.0:1734017174.870671:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017174.870674:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017174.870678:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017174.870679:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017174/real 1734017174] req@ffff8800a73cc380 x1818247332751488/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017174.870685:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332752256/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017174.870700:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332751488/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017174.870712:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017174.870715:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017174.870719:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332751488:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017174.870727:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332751616:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017174.870743:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017174.870750:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017174/real 1734017174] req@ffff8800a73cce00 x1818247332751616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017174.870761:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332751616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017174.870770:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017174.870773:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017174.870777:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332751616:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017174.870783:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332751744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017174.870796:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017174.870802:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017174/real 1734017174] req@ffff8800a73cf100 x1818247332751744/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017174.870812:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332751744/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017174.870819:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017174.870822:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017174.870825:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332751744:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017174.870850:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332751872:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017174.870862:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017174.870869:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017174/real 1734017174] req@ffff8800a73cdc00 x1818247332751872/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017174.870879:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332751872/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017174.870887:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017174.870889:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017174.870892:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332751872:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017174.870898:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332752000:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017174.870910:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017174.870916:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017174/real 1734017174] req@ffff8800a73ce680 x1818247332752000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017174.870926:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332752000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017174.870933:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017174.870936:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017174.870938:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332752000:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017174.870943:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332752128:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017174.870954:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017174.870960:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017174/real 1734017174] req@ffff8800a73cc700 x1818247332752128/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017174.870971:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332752128/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017174.870979:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017174.870981:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017174.870984:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332752128:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017174.870989:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332752256:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017174.871001:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017174.871007:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017174/real 1734017174] req@ffff8800a73cdf80 x1818247332752256/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017174.871018:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332752256/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017174.871026:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017174.871028:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017174.871031:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332752256:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017175.056462:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017176.058487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017177.060477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017178.062498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017179.064485:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017179.878348:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017179.878357:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017179.878361:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017179.878371:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1442 00000100:00080000:3.0:1734017179.878381:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017179.878389:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017179.878425:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332752384/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017179.878460:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017179.878464:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017179.878466:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017179.878471:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1442 00000100:00080000:3.0:1734017179.878474:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017179.878477:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017179.878487:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017179.878499:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332752512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017179.878509:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017179.878512:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017179.878514:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017179.878518:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1442 00000100:00080000:3.0:1734017179.878521:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017179.878525:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017179.878530:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017179.878538:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332752640/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017179.878547:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017179.878550:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017179.878551:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017179.878554:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1442 00000100:00080000:3.0:1734017179.878557:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017179.878560:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017179.878564:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017179.878571:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332752768/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017179.878580:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017179.878583:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1734017179.878585:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332752384:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017179.878585:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017179.878589:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1442 00000100:00080000:3.0:1734017179.878591:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017179.878594:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017179.878598:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017179.878606:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332752896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017179.878621:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017179.878624:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017179.878626:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017179.878630:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1442 00000100:00080000:3.0:1734017179.878632:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017179.878636:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017179.878640:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017179.878647:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332753024/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017179.878651:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017179.878655:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017179.878658:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017179.878660:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017179.878663:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1442 00000100:00080000:3.0:1734017179.878668:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017179.878671:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017179.878672:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017179/real 1734017179] req@ffff8800a73cf800 x1818247332752384/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017179.878675:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017179.878682:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332753152/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017179.878692:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332752384/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017179.878705:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017179.878708:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017179.878713:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332752384:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017179.878720:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332752512:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017179.878738:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017179.878745:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017179/real 1734017179] req@ffff8800a73cca80 x1818247332752512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017179.878756:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332752512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017179.878765:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017179.878768:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017179.878772:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332752512:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017179.878778:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332752640:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017179.878791:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017179.878797:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017179/real 1734017179] req@ffff8800a73cf480 x1818247332752640/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017179.878807:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332752640/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017179.878814:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017179.878817:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017179.878820:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332752640:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017179.878845:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332752768:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017179.878858:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017179.878864:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017179/real 1734017179] req@ffff8800a73cfb80 x1818247332752768/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017179.878875:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332752768/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017179.878882:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017179.878884:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017179.878887:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332752768:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017179.878893:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332752896:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017179.878906:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017179.878911:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017179/real 1734017179] req@ffff8800a73cd500 x1818247332752896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017179.878921:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332752896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017179.878928:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017179.878931:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017179.878933:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332752896:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017179.878938:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332753024:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017179.878949:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017179.879007:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017179/real 1734017179] req@ffff8800a73cea00 x1818247332753024/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017179.879018:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332753024/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017179.879026:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017179.879029:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017179.879032:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332753024:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017179.879037:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332753152:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017179.879054:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017179.879061:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017179/real 1734017179] req@ffff8800a73ced80 x1818247332753152/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017179.879072:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332753152/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017179.879080:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017179.879082:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017179.879085:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332753152:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017180.066508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017181.068464:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017182.070493:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017183.072516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017184.074537:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017184.886339:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017184.886348:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017184.886352:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017184.886363:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1447 00000100:00080000:3.0:1734017184.886373:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017184.886382:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017184.886417:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332753280/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017184.886453:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017184.886457:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017184.886459:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017184.886464:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1447 00000100:00080000:3.0:1734017184.886467:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017184.886471:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017184.886481:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017184.886493:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332753408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017184.886503:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017184.886506:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017184.886508:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017184.886512:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1447 00000100:00080000:3.0:1734017184.886515:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017184.886518:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017184.886522:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017184.886530:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332753536/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017184.886539:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017184.886542:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017184.886544:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017184.886547:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1447 00000100:00080000:3.0:1734017184.886550:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017184.886553:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017184.886558:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017184.886562:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332753280:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017184.886565:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332753664/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017184.886574:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017184.886577:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017184.886579:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017184.886582:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1447 00000100:00080000:3.0:1734017184.886585:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017184.886588:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017184.886592:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017184.886600:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332753792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017184.886608:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017184.886610:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017184.886612:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017184.886616:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1447 00000100:00080000:3.0:1734017184.886618:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017184.886622:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017184.886627:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017184.886629:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017184.886634:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332753920/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017184.886643:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017184.886646:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017184.886647:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017184.886651:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1447 00000100:00100000:1.0:1734017184.886652:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017184/real 1734017184] req@ffff8800a73cd880 x1818247332753280/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017184.886653:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017184.886656:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017184.886660:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017184.886668:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332754048/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017184.886672:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332753280/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017184.886683:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017184.886686:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017184.886691:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332753280:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017184.886698:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332753408:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017184.886715:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017184.886722:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017184/real 1734017184] req@ffff8800a73ced80 x1818247332753408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017184.886734:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332753408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017184.886743:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017184.886746:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017184.886749:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332753408:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017184.886755:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332753536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017184.886769:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017184.886775:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017184/real 1734017184] req@ffff8800a73cea00 x1818247332753536/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017184.886785:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332753536/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017184.886792:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017184.886795:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017184.886798:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332753536:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017184.886823:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332753664:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017184.886836:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017184.886842:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017184/real 1734017184] req@ffff8800a73cd500 x1818247332753664/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017184.886852:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332753664/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017184.886859:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017184.886862:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017184.886865:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332753664:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017184.886922:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332753792:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017184.886940:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017184.886948:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017184/real 1734017184] req@ffff8800a73cfb80 x1818247332753792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017184.886959:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332753792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017184.886967:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017184.886969:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017184.886972:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332753792:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017184.886977:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332753920:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017184.886989:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017184.886995:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017184/real 1734017184] req@ffff8800a73cf480 x1818247332753920/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017184.887005:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332753920/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017184.887012:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017184.887014:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017184.887017:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332753920:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017184.887022:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332754048:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017184.887034:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017184.887040:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017184/real 1734017184] req@ffff8800a73cca80 x1818247332754048/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017184.887050:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332754048/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017184.887058:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017184.887060:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017184.887063:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332754048:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017185.076475:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017186.078481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017187.080444:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017188.082513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017189.084464:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017189.894340:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017189.894349:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017189.894353:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017189.894363:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1452 00000100:00080000:3.0:1734017189.894372:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017189.894380:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017189.894416:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332754176/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017189.894449:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017189.894456:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017189.894458:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017189.894463:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1452 00000100:00080000:3.0:1734017189.894466:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017189.894470:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017189.894479:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017189.894490:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332754304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017189.894500:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017189.894503:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017189.894505:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017189.894509:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1452 00000100:00080000:3.0:1734017189.894511:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017189.894515:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017189.894520:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017189.894528:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332754432/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017189.894536:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017189.894539:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017189.894541:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017189.894544:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1452 00000100:00080000:3.0:1734017189.894547:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017189.894550:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017189.894555:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017189.894563:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332754176:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017189.894563:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332754560/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017189.894571:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017189.894574:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017189.894576:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017189.894579:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1452 00000100:00080000:3.0:1734017189.894582:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017189.894585:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017189.894589:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017189.894596:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332754688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017189.894604:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017189.894607:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017189.894608:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017189.894612:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1452 00000100:00080000:3.0:1734017189.894615:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017189.894618:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017189.894622:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017189.894630:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332754816/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017189.894631:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017189.894638:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017189.894642:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017189.894644:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017189.894647:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1452 00000100:00080000:3.0:1734017189.894650:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017189.894653:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017189/real 1734017189] req@ffff8800a73cf800 x1818247332754176/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017189.894653:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017189.894658:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017189.894666:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332754944/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017189.894673:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332754176/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017189.894684:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017189.894687:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017189.894692:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332754176:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017189.894699:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332754304:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017189.894716:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017189.894722:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017189/real 1734017189] req@ffff8800a73cdf80 x1818247332754304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017189.894733:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332754304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017189.894742:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017189.894744:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017189.894748:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332754304:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017189.894754:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332754432:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017189.894767:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017189.894772:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017189/real 1734017189] req@ffff8800a73cc700 x1818247332754432/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017189.894826:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332754432/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017189.894835:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017189.894838:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017189.894841:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332754432:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017189.894868:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332754560:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017189.894885:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017189.894892:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017189/real 1734017189] req@ffff8800a73ce680 x1818247332754560/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017189.894903:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332754560/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017189.894911:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017189.894913:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017189.894916:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332754560:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017189.894922:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332754688:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017189.894934:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017189.894940:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017189/real 1734017189] req@ffff8800a73cdc00 x1818247332754688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017189.894951:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332754688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017189.894958:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017189.894960:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017189.894963:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332754688:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017189.894968:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332754816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017189.894979:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017189.894985:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017189/real 1734017189] req@ffff8800a73cf100 x1818247332754816/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017189.894995:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332754816/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017189.895002:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017189.895004:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017189.895007:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332754816:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017189.895012:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332754944:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017189.895024:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017189.895030:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017189/real 1734017189] req@ffff8800a73cce00 x1818247332754944/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017189.895040:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332754944/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017189.895048:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017189.895050:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017189.895053:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332754944:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017190.086488:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017191.088476:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017192.090494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017193.092490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017194.094450:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017194.902342:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017194.902350:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017194.902355:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017194.902365:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1457 00000100:00080000:3.0:1734017194.902375:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017194.902383:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017194.902418:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332755072/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017194.902452:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017194.902457:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017194.902459:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017194.902464:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1457 00000100:00080000:3.0:1734017194.902467:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017194.902471:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017194.902480:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017194.902491:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332755200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017194.902502:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017194.902505:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017194.902507:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017194.902510:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1457 00000100:00080000:3.0:1734017194.902513:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017194.902516:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017194.902521:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017194.902529:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332755328/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017194.902537:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017194.902540:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017194.902541:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017194.902545:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1457 00000100:00080000:3.0:1734017194.902547:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017194.902550:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017194.902554:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017194.902562:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332755456/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017194.902568:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332755072:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017194.902571:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017194.902574:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017194.902576:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017194.902580:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1457 00000100:00080000:3.0:1734017194.902583:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017194.902587:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017194.902591:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017194.902599:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332755584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017194.902607:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017194.902610:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017194.902611:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017194.902615:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1457 00000100:00080000:3.0:1734017194.902618:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017194.902621:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017194.902625:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017194.902633:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332755712/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017194.902634:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017194.902640:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017194.902643:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017194.902645:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017194.902648:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1457 00000100:00080000:3.0:1734017194.902651:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017194.902654:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017194.902657:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017194/real 1734017194] req@ffff8800a73cc380 x1818247332755072/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017194.902658:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017194.902666:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332755840/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017194.902676:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332755072/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017194.902687:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017194.902690:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017194.902694:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332755072:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017194.902737:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332755200:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017194.902757:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017194.902765:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017194/real 1734017194] req@ffff8800a73cce00 x1818247332755200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017194.902777:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332755200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017194.902786:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017194.902789:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017194.902792:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332755200:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017194.902799:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332755328:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017194.902812:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017194.902818:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017194/real 1734017194] req@ffff8800a73cf100 x1818247332755328/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017194.902828:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332755328/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017194.902836:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017194.902838:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017194.902842:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332755328:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017194.902867:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332755456:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017194.902880:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017194.902886:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017194/real 1734017194] req@ffff8800a73cdc00 x1818247332755456/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017194.902896:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332755456/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017194.902904:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017194.902906:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017194.902909:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332755456:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017194.902915:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332755584:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017194.902928:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017194.902934:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017194/real 1734017194] req@ffff8800a73ce680 x1818247332755584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017194.902944:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332755584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017194.902952:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017194.902954:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017194.902957:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332755584:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017194.902962:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332755712:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017194.902974:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017194.902979:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017194/real 1734017194] req@ffff8800a73cc700 x1818247332755712/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017194.902989:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332755712/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017194.902996:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017194.902998:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017194.903001:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332755712:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017194.903006:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332755840:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017194.903018:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017194.903024:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017194/real 1734017194] req@ffff8800a73cdf80 x1818247332755840/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017194.903035:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332755840/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017194.903043:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017194.903045:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017194.903048:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332755840:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017195.096449:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017196.098471:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017197.100476:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017198.102489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017199.104492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017199.910336:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017199.910345:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017199.910349:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017199.910360:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1462 00000100:00080000:3.0:1734017199.910370:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017199.910378:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017199.910413:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332755968/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017199.910449:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017199.910453:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017199.910455:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017199.910460:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1462 00000100:00080000:3.0:1734017199.910464:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017199.910467:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017199.910477:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017199.910488:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332756096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017199.910498:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017199.910501:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017199.910503:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017199.910507:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1462 00000100:00080000:3.0:1734017199.910511:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017199.910515:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017199.910520:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017199.910528:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332756224/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017199.910537:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017199.910540:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017199.910541:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017199.910544:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1462 00000100:00080000:3.0:1734017199.910547:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017199.910550:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017199.910554:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017199.910562:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332756352/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017199.910570:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017199.910573:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017199.910575:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017199.910576:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332755968:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017199.910579:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1462 00000100:00080000:3.0:1734017199.910581:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017199.910584:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017199.910588:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017199.910596:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332756480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017199.910641:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017199.910645:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017199.910647:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017199.910652:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1462 00000100:00080000:3.0:1734017199.910655:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017199.910658:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017199.910663:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017199.910672:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332756608/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017199.910681:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734017199.910682:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017199.910684:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017199.910686:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017199.910690:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1462 00000100:00080000:3.0:1734017199.910693:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017199.910696:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017199.910700:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017199.910706:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017199/real 1734017199] req@ffff8800a73cf800 x1818247332755968/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017199.910709:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332756736/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017199.910725:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332755968/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017199.910737:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017199.910740:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017199.910745:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332755968:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017199.910752:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332756096:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017199.910770:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017199.910777:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017199/real 1734017199] req@ffff8800a73cca80 x1818247332756096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017199.910788:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332756096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017199.910796:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017199.910799:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017199.910803:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332756096:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017199.910809:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332756224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017199.910822:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017199.910828:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017199/real 1734017199] req@ffff8800a73cf480 x1818247332756224/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017199.910837:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332756224/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017199.910845:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017199.910847:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017199.910851:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332756224:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017199.910874:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332756352:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017199.910887:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017199.910893:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017199/real 1734017199] req@ffff8800a73cfb80 x1818247332756352/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017199.910903:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332756352/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017199.910911:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017199.910913:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017199.910915:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332756352:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017199.910921:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332756480:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017199.910933:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017199.910939:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017199/real 1734017199] req@ffff8800a73cd500 x1818247332756480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017199.910949:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332756480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017199.910956:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017199.910958:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017199.910961:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332756480:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017199.910966:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332756608:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017199.910977:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017199.910983:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017199/real 1734017199] req@ffff8800a73cea00 x1818247332756608/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017199.910993:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332756608/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017199.911000:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017199.911002:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017199.911005:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332756608:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017199.911010:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332756736:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017199.911022:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017199.911027:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017199/real 1734017199] req@ffff8800a73ced80 x1818247332756736/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017199.911038:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332756736/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017199.911046:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017199.911048:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017199.911050:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332756736:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017200.106440:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017201.108448:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017202.110479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017203.112479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017204.114475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017204.918372:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017204.918382:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017204.918387:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017204.918397:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1467 00000100:00080000:3.0:1734017204.918407:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017204.918415:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017204.918451:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332756864/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017204.918485:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017204.918490:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017204.918492:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017204.918496:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1467 00000100:00080000:3.0:1734017204.918499:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017204.918503:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017204.918513:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017204.918525:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332756992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017204.918583:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017204.918587:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017204.918589:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017204.918593:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1467 00000100:00080000:3.0:1734017204.918596:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017204.918600:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017204.918605:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017204.918615:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332757120/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017204.918620:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332756864:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017204.918625:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017204.918629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017204.918631:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017204.918634:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1467 00000100:00080000:3.0:1734017204.918637:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017204.918640:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017204.918644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017204.918652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332757248/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017204.918660:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017204.918663:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017204.918665:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017204.918668:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1467 00000100:00080000:3.0:1734017204.918672:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017204.918675:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017204.918679:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1734017204.918686:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017204.918687:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332757376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017204.918695:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017204.918698:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017204.918700:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017204.918704:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1467 00000100:00080000:3.0:1734017204.918707:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017204.918708:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017204/real 1734017204] req@ffff8800a73cd880 x1818247332756864/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017204.918711:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017204.918716:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017204.918723:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332757504/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017204.918728:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332756864/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017204.918731:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017204.918734:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017204.918736:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734017204.918740:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017204.918740:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1467 00000100:00080000:3.0:1734017204.918742:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1734017204.918743:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734017204.918745:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017204.918747:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332756864:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017204.918750:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017204.918755:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332756992:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734017204.918758:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332757632/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017204.918772:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017204.918779:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017204/real 1734017204] req@ffff8800a73ced80 x1818247332756992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017204.918790:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332756992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017204.918799:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017204.918802:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017204.918805:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332756992:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017204.918828:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332757120:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017204.918842:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017204.918849:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017204/real 1734017204] req@ffff8800a73cea00 x1818247332757120/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017204.918859:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332757120/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017204.918867:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017204.918869:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017204.918872:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332757120:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017204.918878:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332757248:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017204.918890:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017204.918895:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017204/real 1734017204] req@ffff8800a73cd500 x1818247332757248/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017204.918905:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332757248/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017204.918912:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017204.918914:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017204.918918:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332757248:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017204.918923:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332757376:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017204.918936:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017204.918942:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017204/real 1734017204] req@ffff8800a73cfb80 x1818247332757376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017204.918952:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332757376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017204.918960:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017204.918962:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017204.918965:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332757376:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017204.918970:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332757504:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017204.918982:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017204.918987:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017204/real 1734017204] req@ffff8800a73cf480 x1818247332757504/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017204.918997:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332757504/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017204.919004:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017204.919006:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017204.919009:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332757504:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017204.919014:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332757632:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017204.919026:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017204.919031:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017204/real 1734017204] req@ffff8800a73cca80 x1818247332757632/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017204.919042:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332757632/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017204.919050:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017204.919052:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017204.919055:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332757632:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017205.116502:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017206.118481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017207.120470:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017208.122520:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017209.124546:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017209.926330:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017209.926338:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017209.926343:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017209.926354:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1472 00000100:00080000:3.0:1734017209.926365:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017209.926372:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017209.926408:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332757760/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017209.926474:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017209.926481:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017209.926483:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017209.926488:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1472 00000100:00080000:3.0:1734017209.926492:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017209.926496:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017209.926506:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017209.926518:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332757888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017209.926529:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017209.926532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017209.926534:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017209.926537:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1472 00000100:00080000:3.0:1734017209.926540:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017209.926543:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017209.926548:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017209.926556:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332758016/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017209.926560:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332757760:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017209.926565:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017209.926568:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017209.926570:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017209.926573:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1472 00000100:00080000:3.0:1734017209.926576:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017209.926579:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017209.926584:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017209.926592:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332758144/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017209.926600:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017209.926603:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017209.926605:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017209.926608:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1472 00000100:00080000:3.0:1734017209.926611:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017209.926614:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017209.926618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017209.926626:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332758272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017209.926634:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017209.926637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017209.926638:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017209.926642:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1472 00000100:00080000:3.0:1734017209.926645:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017209.926648:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017209.926652:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017209.926654:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332757888:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734017209.926660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332758400/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017209.926668:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017209.926671:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017209.926673:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017209.926677:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1472 00000100:00080000:3.0:1734017209.926680:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017209.926684:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332758016:192.168.201.152@tcp:8:kworker.0 00000100:00080000:3.0:1734017209.926684:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017209.926688:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017209.926696:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332758528/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017209.926700:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332758144:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017209.926714:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332758272:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017209.926728:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332758400:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017209.926744:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332758528:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017210.126513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017211.128459:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017212.130513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734017212.932467:0:1955:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734017212.932513:0:1955:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734017212.932532:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017212.932567:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017212.932580:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017212.932591:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017212.932601:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017212.932611:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017212.932622:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734017212.932686:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017209/real 1734017212] req@ffff8800a73cce00 x1818247332758528/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017212.932724:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332758528/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017212.932752:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017212.932758:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017212.932765:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332758528:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017212.932777:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017209/real 1734017212] req@ffff8800a73cc700 x1818247332758016/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017212.932797:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332758016/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017212.932810:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017212.932814:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017212.932820:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332758016:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017212.932830:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017209/real 1734017212] req@ffff8800a73ce680 x1818247332758144/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017212.932848:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332758144/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017212.932860:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017212.932864:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017212.932869:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332758144:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017212.932878:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017209/real 1734017212] req@ffff8800a73cdc00 x1818247332758272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017212.932896:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332758272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017212.932918:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017212.932922:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017212.932927:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332758272:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017212.932936:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017209/real 1734017212] req@ffff8800a73cf100 x1818247332758400/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017212.932955:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332758400/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017212.932967:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017212.932972:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017212.932977:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332758400:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017212.932986:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017209/real 1734017212] req@ffff8800a73cf800 x1818247332757760/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017212.933004:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332757760/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017212.933015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017212.933019:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017212.933024:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332757760:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017212.933033:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017209/real 1734017212] req@ffff8800a73cdf80 x1818247332757888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017212.933050:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332757888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017212.933072:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017212.933076:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017212.933082:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332757888:192.168.201.152@tcp:38:kworker.0 00010000:00010000:3.0:1734017213.132470:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017214.134544:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017214.934349:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017214.934393:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017214.934399:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017214.934413:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1477 00000100:00080000:3.0:1734017214.934424:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017214.934434:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017214.934481:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332758656/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017214.934516:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017214.934521:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017214.934523:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017214.934528:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1477 00000100:00080000:3.0:1734017214.934531:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017214.934535:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017214.934547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017214.934561:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332758784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017214.934573:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017214.934576:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017214.934578:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017214.934582:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1477 00000100:00080000:3.0:1734017214.934584:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017214.934588:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017214.934593:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017214.934602:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332758912/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017214.934611:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017214.934614:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017214.934616:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017214.934620:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1477 00000100:00100000:1.0:1734017214.934623:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332758656:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017214.934623:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017214.934626:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017214.934631:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017214.934639:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332759040/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017214.934647:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017214.934651:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017214.934653:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017214.934657:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1477 00000100:00080000:3.0:1734017214.934661:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017214.934665:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017214.934669:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017214.934678:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332759168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017214.934686:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017214.934690:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017214.934691:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017214.934696:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1477 00000100:00080000:3.0:1734017214.934698:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017214.934702:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017214.934706:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017214.934708:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017214.934714:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332759296/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017214.934722:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017214.934726:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017214.934727:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017214.934731:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1477 00000100:00080000:3.0:1734017214.934734:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017214.934735:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017214/real 1734017214] req@ffff8800a73cc380 x1818247332758656/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017214.934737:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017214.934742:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017214.934750:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332759424/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017214.934757:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332758656/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017214.934768:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017214.934771:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017214.934776:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332758656:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017214.934783:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332758784:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017214.934802:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017214.934809:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017214/real 1734017214] req@ffff8800a73cdf80 x1818247332758784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017214.934821:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332758784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017214.934829:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017214.934831:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017214.934835:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332758784:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017214.934863:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332758912:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017214.934879:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017214.934886:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017214/real 1734017214] req@ffff8800a73cf800 x1818247332758912/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017214.934897:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332758912/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017214.934904:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017214.934906:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017214.934909:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332758912:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017214.934914:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332759040:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017214.934926:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017214.934931:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017214/real 1734017214] req@ffff8800a73cf100 x1818247332759040/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017214.934941:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332759040/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017214.934949:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017214.934951:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017214.934954:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332759040:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017214.934959:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332759168:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017214.934972:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017214.934979:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017214/real 1734017214] req@ffff8800a73cdc00 x1818247332759168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017214.934990:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332759168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017214.934997:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017214.934999:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017214.935002:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332759168:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017214.935007:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332759296:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017214.935019:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017214.935025:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017214/real 1734017214] req@ffff8800a73ce680 x1818247332759296/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017214.935034:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332759296/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017214.935041:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017214.935043:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017214.935046:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332759296:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017214.935051:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332759424:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017214.935063:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017214.935069:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017214/real 1734017214] req@ffff8800a73cc700 x1818247332759424/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017214.935079:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332759424/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017214.935094:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017214.935097:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017214.935100:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332759424:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017215.136527:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017216.138541:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017217.140566:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017218.142406:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017219.144552:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017219.942387:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017219.942397:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017219.942403:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017219.942417:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1482 00000100:00080000:3.0:1734017219.942428:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017219.942438:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017219.942488:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332759552/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017219.942526:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017219.942531:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017219.942533:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017219.942538:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1482 00000100:00080000:3.0:1734017219.942542:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017219.942546:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017219.942558:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017219.942569:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332759680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017219.942579:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017219.942583:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017219.942585:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017219.942589:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1482 00000100:00080000:3.0:1734017219.942593:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017219.942597:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017219.942602:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017219.942612:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332759808/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017219.942620:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017219.942623:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017219.942625:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017219.942629:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1482 00000100:00080000:3.0:1734017219.942631:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017219.942635:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017219.942639:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017219.942649:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332759936/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017219.942657:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332759552:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017219.942658:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017219.942662:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017219.942663:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017219.942667:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1482 00000100:00080000:3.0:1734017219.942669:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017219.942673:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017219.942678:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017219.942687:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332760064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017219.942702:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017219.942705:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017219.942707:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017219.942712:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1482 00000100:00080000:3.0:1734017219.942715:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017219.942719:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017219.942723:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017219.942731:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332760192/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017219.942739:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734017219.942740:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017219.942742:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017219.942744:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017219.942747:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1482 00000100:00080000:3.0:1734017219.942750:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017219.942754:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017219.942758:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017219.942765:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017219/real 1734017219] req@ffff8800a73cce00 x1818247332759552/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017219.942768:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332760320/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017219.942785:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332759552/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017219.942797:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017219.942801:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017219.942806:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332759552:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017219.942814:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332759680:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017219.942832:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017219.942839:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017219/real 1734017219] req@ffff8800a73cca80 x1818247332759680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017219.942850:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332759680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017219.942858:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017219.942861:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017219.942866:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332759680:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017219.942873:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332759808:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017219.942887:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017219.942893:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017219/real 1734017219] req@ffff8800a73cf480 x1818247332759808/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017219.942903:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332759808/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017219.942911:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017219.942914:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017219.942917:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332759808:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017219.942944:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332759936:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017219.942957:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017219.942962:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017219/real 1734017219] req@ffff8800a73cfb80 x1818247332759936/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017219.942973:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332759936/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017219.942981:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017219.942983:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017219.942986:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332759936:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017219.942992:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332760064:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017219.943004:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017219.943011:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017219/real 1734017219] req@ffff8800a73cd500 x1818247332760064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017219.943021:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332760064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017219.943029:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017219.943031:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017219.943034:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332760064:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017219.943040:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332760192:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017219.943052:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017219.943058:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017219/real 1734017219] req@ffff8800a73cea00 x1818247332760192/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017219.943068:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332760192/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017219.943075:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017219.943077:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017219.943080:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332760192:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017219.943086:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332760320:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017219.943098:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017219.943104:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017219/real 1734017219] req@ffff8800a73ced80 x1818247332760320/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017219.943114:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332760320/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017219.943129:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017219.943131:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017219.943134:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332760320:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017220.146547:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017221.148511:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017222.150504:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017223.152516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017224.154443:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017224.950383:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017224.950393:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017224.950399:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017224.950414:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1487 00000100:00080000:3.0:1734017224.950427:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017224.950437:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017224.950486:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332760448/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017224.950526:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017224.950530:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017224.950532:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017224.950537:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1487 00000100:00080000:3.0:1734017224.950541:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017224.950546:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017224.950557:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017224.950572:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332760576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017224.950583:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017224.950586:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017224.950588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017224.950593:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1487 00000100:00080000:3.0:1734017224.950596:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017224.950599:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017224.950604:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017224.950613:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332760704/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017224.950622:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017224.950626:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017224.950628:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017224.950631:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1487 00000100:00080000:3.0:1734017224.950634:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017224.950638:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017224.950642:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017224.950650:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332760448:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017224.950652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332760832/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017224.950661:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017224.950665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017224.950666:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017224.950670:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1487 00000100:00080000:3.0:1734017224.950673:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017224.950676:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017224.950681:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017224.950690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332760960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017224.950698:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017224.950701:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017224.950703:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017224.950707:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1487 00000100:00080000:3.0:1734017224.950713:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017224.950717:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734017224.950722:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017224.950722:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017224.950730:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332761088/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017224.950739:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017224.950742:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017224.950744:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017224.950747:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017224/real 1734017224] req@ffff8800a73cd880 x1818247332760448/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017224.950748:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1487 00000100:00080000:3.0:1734017224.950751:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017224.950754:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017224.950758:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017224.950767:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332761216/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017224.950769:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332760448/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017224.950781:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017224.950785:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017224.950790:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332760448:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017224.950798:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332760576:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017224.950817:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017224.950824:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017224/real 1734017224] req@ffff8800a73ced80 x1818247332760576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017224.950836:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332760576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017224.950843:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017224.950847:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017224.950851:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332760576:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017224.950858:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332760704:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017224.950872:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017224.950878:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017224/real 1734017224] req@ffff8800a73cea00 x1818247332760704/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017224.950888:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332760704/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017224.950896:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017224.950899:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017224.950902:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332760704:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017224.950928:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332760832:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017224.950941:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017224.950947:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017224/real 1734017224] req@ffff8800a73cd500 x1818247332760832/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017224.950957:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332760832/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017224.950965:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017224.950968:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017224.950971:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332760832:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017224.950977:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332760960:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017224.950992:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017224.950998:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017224/real 1734017224] req@ffff8800a73cfb80 x1818247332760960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017224.951009:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332760960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017224.951016:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017224.951019:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017224.951021:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332760960:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017224.951027:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332761088:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017224.951039:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017224.951045:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017224/real 1734017224] req@ffff8800a73cf480 x1818247332761088/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017224.951055:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332761088/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017224.951063:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017224.951066:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017224.951069:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332761088:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017224.951075:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332761216:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017224.951087:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017224.951092:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017224/real 1734017224] req@ffff8800a73cca80 x1818247332761216/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017224.951109:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332761216/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017224.951116:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017224.951119:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017224.951121:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332761216:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017225.156506:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017226.158472:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017227.160342:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017228.162538:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017229.164539:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017229.958381:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017229.958391:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017229.958397:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017229.958411:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1492 00000100:00080000:3.0:1734017229.958424:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017229.958433:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017229.958482:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332761344/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017229.958523:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017229.958531:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017229.958533:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017229.958538:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1492 00000100:00080000:3.0:1734017229.958542:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017229.958546:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017229.958559:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017229.958571:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332761472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017229.958583:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017229.958586:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332761344:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017229.958586:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017229.958588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017229.958592:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1492 00000100:00080000:3.0:1734017229.958595:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017229.958599:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017229.958604:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017229.958613:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332761600/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017229.958622:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017229.958625:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017229.958626:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017229.958630:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1492 00000100:00080000:3.0:1734017229.958633:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017229.958636:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017229.958642:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017229.958650:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332761728/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017229.958659:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017229.958659:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017229.958662:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017229.958664:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017229.958668:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1492 00000100:00080000:3.0:1734017229.958671:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017229.958675:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017229.958679:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734017229.958684:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017229/real 1734017229] req@ffff8800a73cce00 x1818247332761344/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017229.958688:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332761856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017229.958697:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017229.958700:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017229.958702:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017229.958705:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332761344/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017229.958707:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1492 00000100:00080000:3.0:1734017229.958709:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017229.958713:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:1.0:1734017229.958717:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1734017229.958717:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734017229.958720:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017229.958726:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332761344:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017229.958727:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332761984/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017229.958735:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017229.958738:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017229.958741:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017229.958745:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1492 00000100:00080000:3.0:1734017229.958748:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017229.958751:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332761472:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017229.958751:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017229.958756:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017229.958768:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332762112/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017229.958769:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017229.958776:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017229/real 1734017229] req@ffff8800a73cc700 x1818247332761472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017229.958788:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332761472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017229.958798:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017229.958801:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017229.958804:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332761472:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017229.958811:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332761600:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017229.958826:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017229.958832:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017229/real 1734017229] req@ffff8800a73ce680 x1818247332761600/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017229.958842:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332761600/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017229.958850:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017229.958852:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017229.958856:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332761600:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017229.958862:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332761728:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017229.958875:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017229.958880:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017229/real 1734017229] req@ffff8800a73cdc00 x1818247332761728/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017229.958890:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332761728/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017229.958898:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017229.958901:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017229.958904:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332761728:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017229.958916:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332761856:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017229.958929:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017229.958935:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017229/real 1734017229] req@ffff8800a73cf100 x1818247332761856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017229.958946:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332761856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017229.958955:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017229.958957:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017229.958960:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332761856:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017229.958966:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332761984:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017229.958978:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017229.958984:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017229/real 1734017229] req@ffff8800a73cf800 x1818247332761984/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017229.958993:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332761984/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017229.959000:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017229.959003:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017229.959006:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332761984:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017229.959024:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332762112:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017229.959037:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017229.959043:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017229/real 1734017229] req@ffff8800a73cdf80 x1818247332762112/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017229.959054:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332762112/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017229.959063:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017229.959065:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017229.959068:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332762112:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017230.166418:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017231.168522:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017232.170488:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017233.172489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017234.174458:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017234.966350:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017234.966360:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017234.966365:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017234.966379:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1497 00000100:00080000:3.0:1734017234.966391:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017234.966401:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017234.966450:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332762240/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017234.966489:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017234.966494:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017234.966496:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017234.966502:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1497 00000100:00080000:3.0:1734017234.966505:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017234.966510:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017234.966521:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017234.966536:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332762368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017234.966548:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017234.966552:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017234.966553:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017234.966557:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1497 00000100:00080000:3.0:1734017234.966560:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017234.966563:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017234.966569:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017234.966578:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332762496/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017234.966587:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017234.966590:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017234.966591:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017234.966595:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1497 00000100:00080000:3.0:1734017234.966597:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017234.966601:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017234.966606:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017234.966615:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332762240:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017234.966615:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332762624/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017234.966625:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017234.966629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017234.966630:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017234.966634:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1497 00000100:00080000:3.0:1734017234.966638:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017234.966642:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017234.966647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017234.966655:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332762752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017234.966664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017234.966667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017234.966668:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017234.966673:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1497 00000100:00080000:3.0:1734017234.966677:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017234.966680:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017234.966685:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017234.966687:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017234.966693:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332762880/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017234.966701:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017234.966704:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017234.966706:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017234.966710:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1497 00000100:00100000:1.0:1734017234.966711:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017234/real 1734017234] req@ffff8800a73cc380 x1818247332762240/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017234.966713:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017234.966716:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017234.966721:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017234.966729:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332763008/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017234.966732:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332762240/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017234.966744:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017234.966748:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017234.966753:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332762240:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017234.966762:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332762368:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017234.966781:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017234.966788:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017234/real 1734017234] req@ffff8800a73cdf80 x1818247332762368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017234.966799:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332762368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017234.966809:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017234.966812:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017234.966816:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332762368:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017234.966823:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332762496:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017234.966837:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017234.966843:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017234/real 1734017234] req@ffff8800a73cf800 x1818247332762496/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017234.966853:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332762496/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017234.966860:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017234.966863:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017234.966867:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332762496:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017234.966893:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332762624:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017234.966906:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017234.966913:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017234/real 1734017234] req@ffff8800a73cf100 x1818247332762624/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017234.966924:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332762624/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017234.966931:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017234.966934:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017234.966937:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332762624:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017234.966944:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332762752:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017234.966957:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017234.966963:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017234/real 1734017234] req@ffff8800a73cdc00 x1818247332762752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017234.966974:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332762752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017234.966981:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017234.966983:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017234.966986:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332762752:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017234.966992:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332762880:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017234.967004:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017234.967010:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017234/real 1734017234] req@ffff8800a73ce680 x1818247332762880/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017234.967070:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332762880/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017234.967079:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017234.967082:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017234.967085:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332762880:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017234.967091:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332763008:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017234.967108:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017234.967114:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017234/real 1734017234] req@ffff8800a73cc700 x1818247332763008/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017234.967126:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332763008/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017234.967134:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017234.967136:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017234.967139:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332763008:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017235.176458:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017236.178442:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017237.180499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017238.182493:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017239.184507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017239.974370:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017239.974379:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017239.974385:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017239.974399:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1502 00000100:00080000:3.0:1734017239.974412:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017239.974421:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017239.974471:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332763136/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017239.974509:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017239.974514:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017239.974516:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017239.974521:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1502 00000100:00080000:3.0:1734017239.974524:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017239.974529:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017239.974541:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017239.974553:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332763264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017239.974563:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017239.974567:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017239.974569:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017239.974573:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1502 00000100:00080000:3.0:1734017239.974577:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017239.974580:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017239.974586:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017239.974595:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332763392/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017239.974604:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017239.974607:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017239.974608:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017239.974612:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1502 00000100:00080000:3.0:1734017239.974615:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017239.974618:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017239.974623:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017239.974633:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332763520/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017239.974634:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332763136:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017239.974643:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017239.974646:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017239.974647:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017239.974651:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1502 00000100:00080000:3.0:1734017239.974654:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017239.974657:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017239.974662:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017239.974671:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332763648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017239.974686:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017239.974689:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017239.974691:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017239.974696:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1502 00000100:00080000:3.0:1734017239.974699:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017239.974702:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734017239.974707:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017239.974707:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017239.974714:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332763776/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017239.974723:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017239.974726:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017239.974728:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017239.974732:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017239/real 1734017239] req@ffff8800a73cce00 x1818247332763136/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017239.974732:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1502 00000100:00080000:3.0:1734017239.974735:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017239.974738:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017239.974742:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017239.974749:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332763904/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017239.974752:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332763136/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017239.974764:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017239.974768:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017239.974773:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332763136:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017239.974782:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332763264:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017239.974800:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017239.974806:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017239/real 1734017239] req@ffff8800a73cca80 x1818247332763264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017239.974818:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332763264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017239.974827:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017239.974830:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017239.974834:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332763264:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017239.974840:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332763392:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017239.974854:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017239.974860:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017239/real 1734017239] req@ffff8800a73cf480 x1818247332763392/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017239.974870:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332763392/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017239.974877:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017239.974880:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017239.974883:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332763392:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017239.974908:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332763520:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017239.974921:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017239.974927:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017239/real 1734017239] req@ffff8800a73cfb80 x1818247332763520/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017239.974985:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332763520/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017239.974993:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017239.974996:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017239.975000:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332763520:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017239.975007:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332763648:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017239.975024:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017239.975031:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017239/real 1734017239] req@ffff8800a73cd500 x1818247332763648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017239.975042:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332763648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017239.975050:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017239.975052:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017239.975055:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332763648:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017239.975062:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332763776:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017239.975074:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017239.975080:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017239/real 1734017239] req@ffff8800a73cea00 x1818247332763776/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017239.975089:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332763776/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017239.975096:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017239.975099:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017239.975102:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332763776:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017239.975108:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332763904:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017239.975121:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017239.975127:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017239/real 1734017239] req@ffff8800a73ced80 x1818247332763904/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017239.975137:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332763904/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017239.975145:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017239.975148:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017239.975150:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332763904:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017240.186495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017241.188499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017242.190495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017243.192522:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017244.194491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017244.982369:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017244.982380:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017244.982386:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017244.982400:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1507 00000100:00080000:3.0:1734017244.982414:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017244.982424:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017244.982473:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332764032/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017244.982512:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017244.982516:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017244.982518:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017244.982524:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1507 00000100:00080000:3.0:1734017244.982527:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017244.982531:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017244.982543:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017244.982557:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332764160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017244.982568:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017244.982571:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017244.982573:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017244.982578:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1507 00000100:00080000:3.0:1734017244.982581:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017244.982584:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017244.982590:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017244.982599:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332764288/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017244.982608:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017244.982612:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017244.982614:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017244.982617:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1507 00000100:00080000:3.0:1734017244.982620:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017244.982624:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017244.982629:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017244.982636:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332764032:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017244.982638:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332764416/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017244.982647:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017244.982651:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017244.982652:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017244.982656:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1507 00000100:00080000:3.0:1734017244.982659:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017244.982663:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017244.982667:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017244.982676:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332764544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017244.982684:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017244.982687:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017244.982689:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017244.982693:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1507 00000100:00080000:3.0:1734017244.982699:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017244.982703:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734017244.982708:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017244.982709:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017244.982717:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332764672/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017244.982726:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017244.982730:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017244.982731:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017244.982732:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017244/real 1734017244] req@ffff8800a73cd880 x1818247332764032/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017244.982735:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1507 00000100:00080000:3.0:1734017244.982738:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017244.982742:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017244.982746:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017244.982753:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332764032/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017244.982755:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332764800/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017244.982765:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017244.982769:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017244.982774:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332764032:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017244.982782:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332764160:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017244.982801:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017244.982808:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017244/real 1734017244] req@ffff8800a73ced80 x1818247332764160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017244.982819:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332764160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017244.982828:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017244.982831:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017244.982834:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332764160:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017244.982841:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332764288:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017244.982906:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017244.982914:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017244/real 1734017244] req@ffff8800a73cea00 x1818247332764288/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017244.982925:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332764288/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017244.982932:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017244.982935:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017244.982938:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332764288:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017244.982965:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332764416:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017244.982978:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017244.982984:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017244/real 1734017244] req@ffff8800a73cd500 x1818247332764416/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017244.982995:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332764416/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017244.983002:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017244.983005:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017244.983008:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332764416:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017244.983015:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332764544:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017244.983028:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017244.983034:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017244/real 1734017244] req@ffff8800a73cfb80 x1818247332764544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017244.983044:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332764544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017244.983052:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017244.983055:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017244.983058:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332764544:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017244.983063:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332764672:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017244.983075:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017244.983081:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017244/real 1734017244] req@ffff8800a73cf480 x1818247332764672/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017244.983091:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332764672/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017244.983098:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017244.983100:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017244.983103:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332764672:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017244.983108:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332764800:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017244.983121:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017244.983127:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017244/real 1734017244] req@ffff8800a73cca80 x1818247332764800/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017244.983138:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332764800/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017244.983146:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017244.983148:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017244.983151:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332764800:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017245.196510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017246.198500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017247.200455:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017248.202455:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017249.204511:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017249.990398:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017249.990408:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017249.990413:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017249.990427:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1512 00000100:00080000:3.0:1734017249.990439:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017249.990449:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017249.990498:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332764928/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017249.990537:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017249.990545:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017249.990547:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017249.990552:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1512 00000100:00080000:3.0:1734017249.990555:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017249.990560:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017249.990571:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017249.990583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332765056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017249.990594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017249.990597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017249.990599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017249.990603:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1512 00000100:00080000:3.0:1734017249.990606:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017249.990611:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017249.990616:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017249.990625:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332765184/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017249.990635:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017249.990638:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017249.990640:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017249.990643:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1512 00000100:00080000:3.0:1734017249.990646:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017249.990650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017249.990655:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332764928:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017249.990656:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017249.990665:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332765312/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017249.990674:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017249.990677:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017249.990678:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017249.990682:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1512 00000100:00080000:3.0:1734017249.990685:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017249.990689:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017249.990693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017249.990702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332765440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017249.990710:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017249.990713:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017249.990715:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017249.990719:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1512 00000100:00080000:3.0:1734017249.990722:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017249.990726:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734017249.990727:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017249.990731:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017249.990739:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332765568/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017249.990748:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017249.990751:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017249/real 1734017249] req@ffff8800a73cce00 x1818247332764928/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017249.990751:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017249.990754:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017249.990788:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1512 00000100:00080000:3.0:1734017249.990792:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017249.990796:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017249.990802:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017249.990804:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332764928/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017249.990813:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332765696/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017249.990816:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017249.990819:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017249.990825:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332764928:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017249.990834:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332765056:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017249.990855:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017249.990862:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017249/real 1734017249] req@ffff8800a73cc700 x1818247332765056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017249.990874:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332765056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017249.990882:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017249.990886:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017249.990889:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332765056:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017249.990914:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332765184:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017249.990928:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017249.990935:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017249/real 1734017249] req@ffff8800a73ce680 x1818247332765184/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017249.990945:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332765184/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017249.990953:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017249.990956:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017249.990959:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332765184:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017249.990965:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332765312:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017249.990977:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017249.990982:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017249/real 1734017249] req@ffff8800a73cdc00 x1818247332765312/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017249.990992:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332765312/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017249.990999:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017249.991002:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017249.991005:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332765312:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017249.991012:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332765440:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017249.991024:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017249.991030:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017249/real 1734017249] req@ffff8800a73cf100 x1818247332765440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017249.991040:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332765440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017249.991048:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017249.991050:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017249.991053:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332765440:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017249.991059:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332765568:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017249.991071:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017249.991077:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017249/real 1734017249] req@ffff8800a73cf800 x1818247332765568/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017249.991087:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332765568/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017249.991094:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017249.991097:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017249.991100:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332765568:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017249.991105:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332765696:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017249.991118:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017249.991124:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017249/real 1734017249] req@ffff8800a73cdf80 x1818247332765696/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017249.991134:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332765696/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017249.991142:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017249.991144:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017249.991147:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332765696:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017250.206452:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017251.208497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017252.210494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017253.212514:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017254.214483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017254.998378:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017254.998389:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017254.998394:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017254.998407:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1517 00000100:00080000:3.0:1734017254.998420:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017254.998430:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017254.998477:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332765824/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017254.998515:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017254.998520:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017254.998522:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017254.998528:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1517 00000100:00080000:3.0:1734017254.998531:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017254.998535:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017254.998547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017254.998562:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332765952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017254.998574:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017254.998578:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017254.998580:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017254.998583:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1517 00000100:00080000:3.0:1734017254.998586:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017254.998590:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017254.998595:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017254.998604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332766080/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017254.998615:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017254.998618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017254.998620:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017254.998623:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1517 00000100:00080000:3.0:1734017254.998626:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017254.998630:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017254.998634:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332765824:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017254.998635:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017254.998644:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332766208/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017254.998653:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017254.998656:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017254.998658:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017254.998661:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1517 00000100:00080000:3.0:1734017254.998666:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017254.998669:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017254.998716:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017254.998728:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332766336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017254.998738:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734017254.998739:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017254.998741:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017254.998743:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017254.998747:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1517 00000100:00080000:3.0:1734017254.998751:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017254.998754:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017254.998759:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017254.998766:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017254/real 1734017254] req@ffff8800a73cc380 x1818247332765824/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017254.998768:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332766464/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017254.998777:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017254.998781:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017254.998782:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017254.998786:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332765824/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017254.998786:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1517 00000100:00080000:3.0:1734017254.998788:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017254.998792:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017254.998796:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734017254.998798:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017254.998802:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017254.998805:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332766592/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017254.998807:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332765824:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017254.998816:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332765952:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017254.998836:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017254.998843:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017254/real 1734017254] req@ffff8800a73cdf80 x1818247332765952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017254.998854:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332765952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017254.998863:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017254.998866:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017254.998869:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332765952:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017254.998894:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332766080:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017254.998909:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017254.998915:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017254/real 1734017254] req@ffff8800a73cf800 x1818247332766080/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017254.998925:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332766080/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017254.998933:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017254.998936:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017254.998939:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332766080:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017254.998945:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332766208:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017254.998956:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017254.998962:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017254/real 1734017254] req@ffff8800a73cf100 x1818247332766208/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017254.998971:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332766208/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017254.998979:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017254.998981:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017254.998984:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332766208:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017254.998990:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332766336:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017254.999003:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017254.999009:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017254/real 1734017254] req@ffff8800a73cdc00 x1818247332766336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017254.999019:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332766336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017254.999027:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017254.999030:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017254.999033:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332766336:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017254.999038:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332766464:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017254.999050:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017254.999056:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017254/real 1734017254] req@ffff8800a73ce680 x1818247332766464/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017254.999066:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332766464/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017254.999072:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017254.999075:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017254.999078:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332766464:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017254.999084:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332766592:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017254.999097:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017254.999102:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017254/real 1734017254] req@ffff8800a73cc700 x1818247332766592/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017254.999113:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332766592/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017254.999121:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017254.999123:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017254.999126:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332766592:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017255.216477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017256.218508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017257.220520:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017258.222538:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017259.224520:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017260.006384:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017260.006394:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017260.006400:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017260.006413:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1522 00000100:00080000:3.0:1734017260.006426:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017260.006436:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017260.006485:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332766720/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017260.006524:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017260.006529:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017260.006531:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017260.006536:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1522 00000100:00080000:3.0:1734017260.006540:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017260.006544:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017260.006556:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017260.006568:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332766848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017260.006578:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017260.006582:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017260.006583:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017260.006628:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1522 00000100:00080000:3.0:1734017260.006634:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017260.006638:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017260.006644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017260.006654:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332766976/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017260.006666:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017260.006669:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017260.006671:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017260.006675:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1522 00000100:00080000:3.0:1734017260.006677:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017260.006681:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017260.006682:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332766720:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017260.006686:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017260.006695:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332767104/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017260.006704:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017260.006707:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017260.006709:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017260.006713:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1522 00000100:00080000:3.0:1734017260.006715:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017260.006719:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017260.006724:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017260.006733:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332767232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017260.006748:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017260.006751:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017260.006753:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734017260.006755:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017260.006758:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1522 00000100:00080000:3.0:1734017260.006761:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017260.006765:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017260.006769:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017260.006777:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332767360/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017260.006779:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017260/real 1734017260] req@ffff8800a73cce00 x1818247332766720/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017260.006785:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017260.006789:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017260.006790:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017260.006794:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1522 00000100:00080000:3.0:1734017260.006797:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017260.006799:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332766720/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017260.006800:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017260.006805:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734017260.006811:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017260.006814:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017260.006815:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332767488/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017260.006820:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332766720:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017260.006829:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332766848:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017260.006847:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017260.006854:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017260/real 1734017260] req@ffff8800a73cca80 x1818247332766848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017260.006865:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332766848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017260.006874:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017260.006877:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017260.006880:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332766848:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017260.006905:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332766976:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017260.006919:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017260.006926:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017260/real 1734017260] req@ffff8800a73cf480 x1818247332766976/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017260.006936:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332766976/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017260.006944:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017260.006946:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017260.006949:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332766976:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017260.006956:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332767104:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017260.006967:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017260.006973:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017260/real 1734017260] req@ffff8800a73cfb80 x1818247332767104/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017260.006983:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332767104/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017260.006990:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017260.006993:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017260.006995:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332767104:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017260.007002:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332767232:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017260.007016:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017260.007022:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017260/real 1734017260] req@ffff8800a73cd500 x1818247332767232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017260.007032:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332767232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017260.007039:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017260.007041:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017260.007044:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332767232:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017260.007050:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332767360:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017260.007063:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017260.007068:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017260/real 1734017260] req@ffff8800a73cea00 x1818247332767360/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017260.007078:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332767360/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017260.007085:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017260.007087:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017260.007090:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332767360:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017260.007096:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332767488:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017260.007109:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017260.007115:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017260/real 1734017260] req@ffff8800a73ced80 x1818247332767488/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017260.007125:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332767488/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017260.007133:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017260.007135:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017260.007138:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332767488:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017260.226472:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017261.228517:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017262.230471:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017263.232467:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017264.234468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017265.014367:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017265.014378:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017265.014383:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017265.014397:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1527 00000100:00080000:3.0:1734017265.014409:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017265.014419:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017265.014468:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332767616/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017265.014537:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017265.014542:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017265.014544:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017265.014550:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1527 00000100:00080000:3.0:1734017265.014553:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017265.014558:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017265.014570:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017265.014586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332767744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017265.014598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017265.014601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017265.014603:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017265.014607:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1527 00000100:00080000:3.0:1734017265.014609:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017265.014613:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017265.014618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017265.014627:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332767872/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017265.014637:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332767616:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017265.014637:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017265.014641:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017265.014643:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017265.014647:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1527 00000100:00080000:3.0:1734017265.014650:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017265.014653:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017265.014658:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017265.014666:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332768000/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017265.014674:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017265.014678:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017265.014679:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017265.014683:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1527 00000100:00080000:3.0:1734017265.014686:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017265.014690:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017265.014694:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017265.014703:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332768128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017265.014707:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017265.014711:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017265.014714:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017265.014716:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017265.014720:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1527 00000100:00080000:3.0:1734017265.014727:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017265.014731:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017265.014733:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017265/real 1734017265] req@ffff8800a73cd880 x1818247332767616/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017265.014737:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017265.014746:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332768256/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017265.014753:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332767616/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017265.014755:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017265.014758:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017265.014760:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017265.014763:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1527 00000100:00080000:1.0:1734017265.014765:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017265.014766:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1734017265.014769:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734017265.014769:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017265.014773:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017265.014774:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332767616:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017265.014783:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332767744:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734017265.014783:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332768384/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017265.014801:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017265.014808:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017265/real 1734017265] req@ffff8800a73ced80 x1818247332767744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017265.014820:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332767744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017265.014828:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017265.014832:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017265.014835:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332767744:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017265.014860:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332767872:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017265.014874:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017265.014881:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017265/real 1734017265] req@ffff8800a73cea00 x1818247332767872/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017265.014891:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332767872/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017265.014899:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017265.014902:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017265.014905:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332767872:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017265.014911:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332768000:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017265.014923:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017265.014929:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017265/real 1734017265] req@ffff8800a73cd500 x1818247332768000/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017265.014939:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332768000/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017265.014946:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017265.014948:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017265.014951:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332768000:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017265.014957:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332768128:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017265.014969:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017265.014975:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017265/real 1734017265] req@ffff8800a73cfb80 x1818247332768128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017265.014985:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332768128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017265.014992:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017265.014995:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017265.014998:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332768128:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017265.015004:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332768256:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017265.015016:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017265.015022:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017265/real 1734017265] req@ffff8800a73cf480 x1818247332768256/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017265.015032:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332768256/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017265.015039:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017265.015042:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017265.015045:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332768256:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017265.015051:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332768384:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017265.015063:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017265.015068:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017265/real 1734017265] req@ffff8800a73cca80 x1818247332768384/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017265.015079:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332768384/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017265.015087:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017265.015090:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017265.015092:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332768384:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017265.236490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017266.238552:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017267.240532:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017268.242514:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017269.244535:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017270.022393:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017270.022404:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017270.022409:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017270.022463:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1532 00000100:00080000:3.0:1734017270.022476:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017270.022486:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017270.022536:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332768512/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017270.022575:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017270.022583:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017270.022585:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017270.022591:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1532 00000100:00080000:3.0:1734017270.022594:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017270.022598:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017270.022611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017270.022623:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332768640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017270.022633:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017270.022637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017270.022639:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017270.022643:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1532 00000100:00080000:3.0:1734017270.022646:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017270.022650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017270.022656:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017270.022664:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332768768/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017270.022673:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017270.022676:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017270.022678:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017270.022681:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1532 00000100:00080000:3.0:1734017270.022684:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017270.022687:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017270.022694:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017270.022702:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332768512:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017270.022703:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332768896/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017270.022713:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017270.022716:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017270.022718:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017270.022721:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1532 00000100:00080000:3.0:1734017270.022725:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017270.022728:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017270.022733:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017270.022742:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332769024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017270.022751:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017270.022754:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017270.022756:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017270.022761:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1532 00000100:00080000:3.0:1734017270.022764:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017270.022768:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734017270.022772:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017270.022772:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017270.022780:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332769152/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017270.022789:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017270.022792:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017270.022794:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017270.022797:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017270/real 1734017270] req@ffff8800a73cce00 x1818247332768512/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017270.022799:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1532 00000100:00080000:3.0:1734017270.022802:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017270.022805:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017270.022810:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017270.022817:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332768512/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017270.022821:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332769280/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017270.022829:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017270.022833:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017270.022838:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332768512:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017270.022847:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332768640:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017270.022865:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017270.022871:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017270/real 1734017270] req@ffff8800a73cc700 x1818247332768640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017270.022882:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332768640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017270.022891:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017270.022894:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017270.022897:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332768640:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017270.022904:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332768768:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017270.022917:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017270.022923:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017270/real 1734017270] req@ffff8800a73ce680 x1818247332768768/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017270.022934:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332768768/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017270.022941:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017270.022944:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017270.022947:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332768768:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017270.022973:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332768896:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017270.022986:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017270.022992:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017270/real 1734017270] req@ffff8800a73cdc00 x1818247332768896/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017270.023003:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332768896/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017270.023010:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017270.023013:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017270.023016:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332768896:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017270.023022:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332769024:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017270.023035:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017270.023041:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017270/real 1734017270] req@ffff8800a73cf100 x1818247332769024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017270.023052:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332769024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017270.023060:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017270.023063:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017270.023065:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332769024:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017270.023072:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332769152:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017270.023084:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017270.023089:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017270/real 1734017270] req@ffff8800a73cf800 x1818247332769152/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017270.023099:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332769152/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017270.023106:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017270.023109:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017270.023111:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332769152:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017270.023117:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332769280:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017270.023130:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017270.023135:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017270/real 1734017270] req@ffff8800a73cdf80 x1818247332769280/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017270.023145:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332769280/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017270.023153:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017270.023155:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017270.023158:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332769280:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017270.246487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017271.248499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017272.250480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017273.252553:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017274.254508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017275.030407:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017275.030419:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017275.030425:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017275.030439:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1537 00000100:00080000:3.0:1734017275.030451:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017275.030462:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017275.030510:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332769408/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017275.030547:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017275.030552:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017275.030554:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017275.030560:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1537 00000100:00080000:3.0:1734017275.030563:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017275.030568:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017275.030579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017275.030594:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332769536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017275.030607:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017275.030610:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017275.030612:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017275.030616:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1537 00000100:00080000:3.0:1734017275.030618:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017275.030622:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017275.030627:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017275.030636:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332769664/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017275.030645:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017275.030648:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017275.030649:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017275.030653:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1537 00000100:00080000:3.0:1734017275.030655:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017275.030659:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017275.030664:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017275.030672:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332769408:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017275.030673:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332769792/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017275.030683:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017275.030686:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017275.030688:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017275.030693:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1537 00000100:00080000:3.0:1734017275.030697:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017275.030701:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017275.030706:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017275.030714:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332769920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017275.030723:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017275.030726:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017275.030727:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017275.030732:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1537 00000100:00080000:3.0:1734017275.030736:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017275.030740:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017275.030744:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017275.030753:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332770048/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017275.030760:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017275.030764:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017275.030765:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017275.030769:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1537 00000100:00100000:1.0:1734017275.030772:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332769536:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017275.030772:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017275.030775:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017275.030780:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017275.030790:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332770176/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017275.030797:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332769664:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017275.030822:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332769792:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017275.030837:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332769920:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017275.030854:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332770048:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017275.030870:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332770176:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017275.256504:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017276.258531:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017277.260400:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734017278.036534:0:1957:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734017278.036584:0:1957:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734017278.036606:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017278.036640:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017278.036648:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017278.036655:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017278.036661:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017278.036667:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017278.036674:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734017278.036745:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017275/real 1734017278] req@ffff8800a73cf100 x1818247332769792/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017278.036775:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332769792/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017278.036795:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017278.036799:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017278.036804:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332769792:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017278.036812:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017275/real 1734017278] req@ffff8800a73cdc00 x1818247332769920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017278.036834:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332769920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017278.036842:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017278.036845:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017278.036849:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332769920:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017278.036856:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017275/real 1734017278] req@ffff8800a73ce680 x1818247332770048/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017278.036866:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332770048/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017278.036873:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017278.036877:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017278.036881:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332770048:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017278.036891:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017275/real 1734017278] req@ffff8800a73cc700 x1818247332770176/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017278.036908:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332770176/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017278.036929:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017278.036933:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017278.036938:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332770176:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017278.036949:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017275/real 1734017278] req@ffff8800a73cc380 x1818247332769408/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017278.036968:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332769408/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017278.036980:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017278.036984:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017278.036989:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332769408:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017278.036999:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017275/real 1734017278] req@ffff8800a73cdf80 x1818247332769536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017278.037018:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332769536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017278.037030:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017278.037034:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017278.037040:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332769536:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017278.037049:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017275/real 1734017278] req@ffff8800a73cf800 x1818247332769664/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017278.037068:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332769664/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017278.037088:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017278.037093:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017278.037098:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332769664:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017278.262483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017279.264520:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017280.038396:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017280.038406:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017280.038412:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017280.038425:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1542 00000100:00080000:3.0:1734017280.038438:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017280.038448:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017280.038498:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332770304/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017280.038537:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017280.038541:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017280.038543:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017280.038548:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1542 00000100:00080000:3.0:1734017280.038552:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017280.038556:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017280.038568:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017280.038579:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332770432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017280.038589:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017280.038593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017280.038595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017280.038599:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1542 00000100:00080000:3.0:1734017280.038603:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017280.038607:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017280.038612:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017280.038622:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332770560/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017280.038631:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017280.038634:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017280.038636:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017280.038639:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1542 00000100:00080000:3.0:1734017280.038642:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017280.038645:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017280.038650:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017280.038658:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332770688/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017280.038668:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017280.038671:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332770304:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017280.038671:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017280.038672:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017280.038677:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1542 00000100:00080000:3.0:1734017280.038679:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017280.038683:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017280.038687:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017280.038697:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332770816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017280.038713:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017280.038717:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017280.038718:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017280.038723:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1542 00000100:00080000:3.0:1734017280.038726:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017280.038730:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017280.038734:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017280.038742:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017280.038742:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332770944/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017280.038750:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017280.038753:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017280.038755:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017280.038759:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1542 00000100:00080000:3.0:1734017280.038762:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017280.038764:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017280/real 1734017280] req@ffff8800a73cce00 x1818247332770304/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017280.038765:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017280.038769:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017280.038780:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332771072/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017280.038784:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332770304/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017280.038796:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017280.038800:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017280.038804:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332770304:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017280.038813:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332770432:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017280.038832:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017280.038839:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017280/real 1734017280] req@ffff8800a73cca80 x1818247332770432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017280.038850:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332770432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017280.038858:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017280.038861:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017280.038865:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332770432:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017280.038871:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332770560:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017280.038885:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017280.038891:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017280/real 1734017280] req@ffff8800a73cf480 x1818247332770560/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017280.038902:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332770560/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017280.038909:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017280.038912:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017280.038915:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332770560:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017280.038940:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332770688:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017280.038953:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017280.038959:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017280/real 1734017280] req@ffff8800a73cfb80 x1818247332770688/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017280.038969:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332770688/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017280.038977:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017280.038979:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017280.038982:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332770688:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017280.038988:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332770816:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017280.039001:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017280.039007:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017280/real 1734017280] req@ffff8800a73cd500 x1818247332770816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017280.039018:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332770816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017280.039025:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017280.039028:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017280.039031:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332770816:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017280.039037:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332770944:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017280.039049:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017280.039055:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017280/real 1734017280] req@ffff8800a73cea00 x1818247332770944/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017280.039065:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332770944/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017280.039072:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017280.039074:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017280.039077:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332770944:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017280.039083:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332771072:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017280.039095:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017280.039101:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017280/real 1734017280] req@ffff8800a73ced80 x1818247332771072/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017280.039111:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332771072/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017280.039126:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017280.039128:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017280.039131:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332771072:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017280.266452:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017281.268500:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017282.270478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017283.272434:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017284.274506:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017285.046356:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017285.046365:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017285.046370:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017285.046380:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1547 00000100:00080000:3.0:1734017285.046390:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017285.046398:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017285.046433:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332771200/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017285.046467:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017285.046472:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017285.046474:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017285.046479:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1547 00000100:00080000:3.0:1734017285.046482:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017285.046486:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017285.046495:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017285.046508:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332771328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017285.046518:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017285.046521:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017285.046523:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017285.046528:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1547 00000100:00080000:3.0:1734017285.046531:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017285.046534:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017285.046539:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017285.046547:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332771456/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017285.046555:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017285.046559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017285.046561:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017285.046565:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1547 00000100:00080000:3.0:1734017285.046568:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017285.046571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017285.046573:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332771200:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017285.046576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017285.046583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332771584/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017285.046592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017285.046595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017285.046597:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017285.046600:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1547 00000100:00080000:3.0:1734017285.046603:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017285.046606:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017285.046610:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017285.046618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332771712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017285.046626:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017285.046629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017285.046631:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017285.046634:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1547 00000100:00080000:3.0:1734017285.046637:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017285.046640:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017285.046641:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017285.046646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017285.046653:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332771840/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017285.046662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017285.046663:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017285/real 1734017285] req@ffff8800a73cd880 x1818247332771200/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017285.046665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017285.046666:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017285.046670:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1547 00000100:00080000:3.0:1734017285.046672:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017285.046676:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017285.046680:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017285.046683:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332771200/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017285.046688:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332771968/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017285.046694:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017285.046698:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017285.046702:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332771200:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017285.046710:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332771328:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017285.046727:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017285.046734:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017285/real 1734017285] req@ffff8800a73ced80 x1818247332771328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017285.046745:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332771328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017285.046752:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017285.046756:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017285.046760:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332771328:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017285.046766:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332771456:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017285.046779:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017285.046785:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017285/real 1734017285] req@ffff8800a73cea00 x1818247332771456/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017285.046795:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332771456/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017285.046803:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017285.046805:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017285.046808:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332771456:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017285.046837:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332771584:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017285.046850:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017285.046855:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017285/real 1734017285] req@ffff8800a73cd500 x1818247332771584/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017285.046866:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332771584/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017285.046874:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017285.046876:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017285.046879:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332771584:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017285.046884:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332771712:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017285.046899:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017285.046905:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017285/real 1734017285] req@ffff8800a73cfb80 x1818247332771712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017285.046915:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332771712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017285.046922:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017285.046925:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017285.046928:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332771712:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017285.046932:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332771840:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017285.046943:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017285.046949:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017285/real 1734017285] req@ffff8800a73cf480 x1818247332771840/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017285.046959:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332771840/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017285.046967:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017285.046969:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017285.046972:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332771840:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017285.046977:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332771968:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017285.046989:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017285.046994:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017285/real 1734017285] req@ffff8800a73cca80 x1818247332771968/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017285.047010:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332771968/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017285.047018:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017285.047020:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017285.047023:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332771968:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017285.276468:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017286.278465:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017287.280454:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017288.282463:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017289.284471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017290.054358:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017290.054366:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017290.054371:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017290.054381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1552 00000100:00080000:3.0:1734017290.054390:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017290.054398:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017290.054433:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332772096/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017290.054467:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017290.054474:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017290.054476:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017290.054481:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1552 00000100:00080000:3.0:1734017290.054484:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017290.054488:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017290.054497:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017290.054507:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332772224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017290.054517:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017290.054521:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017290.054522:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017290.054526:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1552 00000100:00080000:3.0:1734017290.054529:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017290.054533:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017290.054538:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017290.054546:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332772352/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017290.054554:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017290.054557:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017290.054559:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017290.054562:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1552 00000100:00080000:3.0:1734017290.054564:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017290.054568:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017290.054573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017290.054577:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332772096:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017290.054581:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332772480/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017290.054590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017290.054593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017290.054595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017290.054598:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1552 00000100:00080000:3.0:1734017290.054600:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017290.054604:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017290.054608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017290.054615:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332772608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017290.054623:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017290.054626:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017290.054628:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017290.054632:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1552 00000100:00080000:3.0:1734017290.054635:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017290.054638:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017290.054642:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017290.054645:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017290.054650:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332772736/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017290.054657:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017290.054660:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017290.054663:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017290.054667:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017290/real 1734017290] req@ffff8800a73cce00 x1818247332772096/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017290.054667:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1552 00000100:00080000:3.0:1734017290.054670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017290.054673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017290.054678:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017290.054686:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332772864/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017290.054687:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332772096/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017290.054698:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017290.054701:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017290.054706:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332772096:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017290.054713:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332772224:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017290.054730:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017290.054737:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017290/real 1734017290] req@ffff8800a73cf800 x1818247332772224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017290.054748:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332772224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017290.054756:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017290.054759:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017290.054763:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332772224:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017290.054769:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332772352:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017290.054782:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017290.054788:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017290/real 1734017290] req@ffff8800a73cdf80 x1818247332772352/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017290.054798:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332772352/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017290.054806:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017290.054808:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017290.054811:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332772352:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017290.054836:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332772480:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017290.054849:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017290.054855:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017290/real 1734017290] req@ffff8800a73cc380 x1818247332772480/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017290.054865:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332772480/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017290.054872:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017290.054875:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017290.054878:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332772480:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017290.054883:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332772608:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017290.054895:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017290.054901:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017290/real 1734017290] req@ffff8800a73cc700 x1818247332772608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017290.054911:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332772608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017290.054919:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017290.054921:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017290.054924:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332772608:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017290.054929:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332772736:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017290.054940:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017290.054946:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017290/real 1734017290] req@ffff8800a73ce680 x1818247332772736/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017290.054955:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332772736/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017290.054962:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017290.054964:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017290.054967:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332772736:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017290.054972:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332772864:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017290.054984:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017290.054989:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017290/real 1734017290] req@ffff8800a73cdc00 x1818247332772864/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017290.055000:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332772864/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017290.055008:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017290.055010:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017290.055013:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332772864:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017290.286410:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017291.288427:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017292.290494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017293.292493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017294.294453:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017295.062360:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017295.062369:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017295.062374:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017295.062383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1557 00000100:00080000:3.0:1734017295.062393:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017295.062401:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017295.062437:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332772992/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017295.062473:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017295.062477:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017295.062479:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017295.062484:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1557 00000100:00080000:3.0:1734017295.062487:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017295.062491:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017295.062501:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017295.062513:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332773120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017295.062524:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017295.062528:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017295.062530:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017295.062533:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1557 00000100:00080000:3.0:1734017295.062535:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017295.062539:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017295.062543:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017295.062551:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332773248/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017295.062560:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017295.062563:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017295.062564:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017295.062568:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1557 00000100:00080000:3.0:1734017295.062570:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017295.062574:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017295.062578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017295.062585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332773376/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017295.062592:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332772992:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017295.062594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017295.062598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017295.062599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017295.062603:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1557 00000100:00080000:3.0:1734017295.062606:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017295.062610:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017295.062614:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017295.062622:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332773504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017295.062630:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017295.062633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017295.062634:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017295.062638:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1557 00000100:00080000:3.0:1734017295.062641:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017295.062644:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017295.062648:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017295.062655:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332773632/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017295.062660:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017295.062663:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017295.062666:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017295.062668:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017295.062671:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1557 00000100:00100000:1.0:1734017295.062682:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017295/real 1734017295] req@ffff8800a73cf100 x1818247332772992/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017295.062683:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017295.062687:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017295.062692:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017295.062700:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332772992/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017295.062700:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332773760/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017295.062711:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017295.062715:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017295.062719:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332772992:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017295.062726:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332773120:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017295.062743:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017295.062750:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017295/real 1734017295] req@ffff8800a73cdc00 x1818247332773120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017295.062762:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332773120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017295.062770:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017295.062773:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017295.062777:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332773120:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017295.062782:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332773248:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017295.062796:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017295.062801:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017295/real 1734017295] req@ffff8800a73ce680 x1818247332773248/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017295.062812:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332773248/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017295.062819:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017295.062821:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017295.062824:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332773248:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017295.062849:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332773376:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017295.062861:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017295.062867:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017295/real 1734017295] req@ffff8800a73cc700 x1818247332773376/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017295.062878:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332773376/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017295.062885:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017295.062887:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017295.062890:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332773376:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017295.062895:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332773504:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017295.062908:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017295.062914:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017295/real 1734017295] req@ffff8800a73cc380 x1818247332773504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017295.062924:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332773504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017295.062932:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017295.062934:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017295.062937:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332773504:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017295.062942:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332773632:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017295.062953:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017295.062958:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017295/real 1734017295] req@ffff8800a73cdf80 x1818247332773632/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017295.062968:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332773632/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017295.062975:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017295.062977:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017295.062980:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332773632:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017295.062985:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332773760:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017295.063056:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017295.063064:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017295/real 1734017295] req@ffff8800a73cf800 x1818247332773760/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017295.063076:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332773760/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017295.063084:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017295.063086:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017295.063089:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332773760:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017295.296420:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017296.298461:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017297.300476:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017298.302523:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017299.304500:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017300.070360:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017300.070369:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017300.070373:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017300.070383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1562 00000100:00080000:3.0:1734017300.070393:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017300.070400:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017300.070435:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332773888/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017300.070470:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017300.070474:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017300.070476:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017300.070481:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1562 00000100:00080000:3.0:1734017300.070484:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017300.070488:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017300.070498:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017300.070509:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332774016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017300.070519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017300.070522:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017300.070524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017300.070527:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1562 00000100:00080000:3.0:1734017300.070531:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017300.070535:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017300.070539:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017300.070548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332774144/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017300.070556:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017300.070559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017300.070561:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017300.070564:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1562 00000100:00080000:3.0:1734017300.070566:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017300.070569:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017300.070573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017300.070581:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332774272/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017300.070588:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332773888:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017300.070590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017300.070593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017300.070594:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017300.070598:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1562 00000100:00080000:3.0:1734017300.070600:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017300.070604:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017300.070608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017300.070615:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332774400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017300.070630:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017300.070633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017300.070635:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017300.070638:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1562 00000100:00080000:3.0:1734017300.070650:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017300.070654:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734017300.070655:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017300.070659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017300.070666:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332774528/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017300.070675:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017300.070677:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017300/real 1734017300] req@ffff8800a73cce00 x1818247332773888/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017300.070678:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017300.070680:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017300.070683:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1562 00000100:00080000:3.0:1734017300.070686:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017300.070689:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017300.070694:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017300.070697:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332773888/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017300.070703:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332774656/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017300.070707:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017300.070711:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017300.070716:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332773888:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017300.070723:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332774016:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017300.070740:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017300.070747:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017300/real 1734017300] req@ffff8800a73cca80 x1818247332774016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017300.070758:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332774016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017300.070766:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017300.070769:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017300.070772:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332774016:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017300.070778:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332774144:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017300.070792:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017300.070798:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017300/real 1734017300] req@ffff8800a73cf480 x1818247332774144/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017300.070808:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332774144/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017300.070815:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017300.070818:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017300.070821:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332774144:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017300.070845:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332774272:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017300.070858:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017300.070864:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017300/real 1734017300] req@ffff8800a73cfb80 x1818247332774272/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017300.070875:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332774272/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017300.070882:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017300.070885:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017300.070888:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332774272:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017300.070894:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332774400:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017300.070955:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017300.070964:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017300/real 1734017300] req@ffff8800a73cd500 x1818247332774400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017300.070975:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332774400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017300.070984:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017300.070986:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017300.070989:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332774400:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017300.070995:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332774528:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017300.071011:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017300.071017:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017300/real 1734017300] req@ffff8800a73cea00 x1818247332774528/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017300.071028:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332774528/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017300.071035:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017300.071037:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017300.071040:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332774528:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017300.071045:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332774656:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017300.071057:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017300.071062:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017300/real 1734017300] req@ffff8800a73ced80 x1818247332774656/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017300.071073:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332774656/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017300.071081:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017300.071083:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017300.071086:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332774656:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017300.306499:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017301.308445:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017302.310484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017303.312398:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017304.314508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017305.078347:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017305.078356:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017305.078361:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017305.078372:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1567 00000100:00080000:3.0:1734017305.078381:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017305.078389:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017305.078423:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332774784/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017305.078456:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017305.078461:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017305.078463:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017305.078467:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1567 00000100:00080000:3.0:1734017305.078471:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017305.078475:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017305.078484:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017305.078496:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332774912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017305.078506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017305.078509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017305.078511:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017305.078515:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1567 00000100:00080000:3.0:1734017305.078517:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017305.078521:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017305.078525:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017305.078533:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332775040/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017305.078541:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017305.078545:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017305.078547:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017305.078551:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1567 00000100:00080000:3.0:1734017305.078553:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017305.078557:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017305.078561:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017305.078566:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332774784:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017305.078569:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332775168/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017305.078577:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017305.078581:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017305.078582:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017305.078585:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1567 00000100:00080000:3.0:1734017305.078588:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017305.078591:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017305.078595:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017305.078603:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332775296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017305.078611:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017305.078614:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017305.078615:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017305.078619:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1567 00000100:00080000:3.0:1734017305.078622:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017305.078625:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017305.078630:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017305.078632:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017305.078638:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332775424/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017305.078646:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017305.078649:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017305.078651:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017305.078654:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017305/real 1734017305] req@ffff8800a73cd880 x1818247332774784/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017305.078654:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1567 00000100:00080000:3.0:1734017305.078657:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017305.078660:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017305.078664:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017305.078672:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332775552/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017305.078674:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332774784/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017305.078684:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017305.078688:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017305.078692:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332774784:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017305.078699:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332774912:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017305.078716:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017305.078723:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017305/real 1734017305] req@ffff8800a73ced80 x1818247332774912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017305.078734:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332774912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017305.078742:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017305.078745:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017305.078749:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332774912:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017305.078755:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332775040:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017305.078768:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017305.078773:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017305/real 1734017305] req@ffff8800a73cea00 x1818247332775040/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017305.078783:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332775040/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017305.078791:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017305.078793:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017305.078796:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332775040:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017305.078861:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332775168:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017305.078880:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017305.078888:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017305/real 1734017305] req@ffff8800a73cd500 x1818247332775168/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017305.078899:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332775168/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017305.078907:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017305.078909:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017305.078912:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332775168:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017305.078919:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332775296:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017305.078932:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017305.078938:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017305/real 1734017305] req@ffff8800a73cfb80 x1818247332775296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017305.078949:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332775296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017305.078957:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017305.078959:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017305.078962:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332775296:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017305.078967:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332775424:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017305.078979:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017305.078984:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017305/real 1734017305] req@ffff8800a73cf480 x1818247332775424/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017305.078994:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332775424/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017305.079001:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017305.079003:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017305.079006:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332775424:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017305.079011:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332775552:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017305.079023:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017305.079029:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017305/real 1734017305] req@ffff8800a73cca80 x1818247332775552/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017305.079040:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332775552/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017305.079048:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017305.079050:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017305.079052:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332775552:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017305.316432:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017306.318454:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017307.320468:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017308.322493:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00020000:2.0:1734017309.126336:0:4736:0:(osc_cache.c:908:osc_extent_wait()) extent ffff8800aa7e2600@{[389 -> 390/1023], [4|0|+|rpc|wiuY|ffff8800a86f29c0], [32768|2|+|+|ffff88012dbbd8c0|1024|ffff8800a975e660]} lustre-OST0000-osc-ffff8800b5805800: wait ext to 0 timedout, recovery in progress? 00000008:00020000:2.0:1734017309.138069:0:4736:0:(osc_cache.c:908:osc_extent_wait()) ### extent: ffff8800aa7e2600 ns: lustre-OST0000-osc-ffff8800b5805800 lock: ffff88012dbbd8c0/0x37cab3886ce5d618 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1593344->1597439) gid 0 flags: 0x800029400020000 nid: local remote: 0xb46129d8d3178a58 expref: -99 pid: 10366 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1734017309.324431:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017310.086370:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017310.086379:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017310.086384:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017310.086394:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1572 00000100:00080000:3.0:1734017310.086403:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017310.086410:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017310.086445:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332775680/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017310.086479:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017310.086486:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017310.086489:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017310.086493:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1572 00000100:00080000:3.0:1734017310.086496:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017310.086500:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017310.086509:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017310.086520:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332775808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017310.086529:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017310.086532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017310.086534:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017310.086538:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1572 00000100:00080000:3.0:1734017310.086540:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017310.086544:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017310.086548:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017310.086556:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332775936/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017310.086565:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017310.086568:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017310.086569:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017310.086572:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1572 00000100:00080000:3.0:1734017310.086575:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017310.086578:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017310.086583:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017310.086590:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332775680:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017310.086591:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332776064/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017310.086600:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017310.086603:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017310.086605:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017310.086608:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1572 00000100:00080000:3.0:1734017310.086611:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017310.086614:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017310.086618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017310.086625:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332776192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017310.086633:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017310.086636:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017310.086638:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017310.086641:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1572 00000100:00080000:3.0:1734017310.086653:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017310.086657:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017310.086657:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017310.086662:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017310.086670:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332776320/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017310.086678:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017310.086679:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017310/real 1734017310] req@ffff8800a73cce00 x1818247332775680/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017310.086681:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017310.086684:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017310.086687:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1572 00000100:00080000:3.0:1734017310.086690:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017310.086693:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017310.086698:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017310.086699:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332775680/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017310.086707:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332776448/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017310.086709:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017310.086713:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017310.086718:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332775680:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017310.086725:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332775808:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017310.086779:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017310.086788:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017310/real 1734017310] req@ffff8800a73cf800 x1818247332775808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017310.086799:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332775808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017310.086808:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017310.086811:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017310.086814:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332775808:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017310.086820:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332775936:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017310.086835:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017310.086841:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017310/real 1734017310] req@ffff8800a73cdf80 x1818247332775936/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017310.086851:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332775936/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017310.086859:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017310.086861:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017310.086864:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332775936:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017310.086889:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332776064:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017310.086901:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017310.086907:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017310/real 1734017310] req@ffff8800a73cc380 x1818247332776064/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017310.086917:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332776064/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017310.086925:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017310.086927:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017310.086931:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332776064:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017310.086937:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332776192:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017310.086949:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017310.086955:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017310/real 1734017310] req@ffff8800a73cc700 x1818247332776192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017310.086965:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332776192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017310.086973:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017310.086975:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017310.086978:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332776192:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017310.086983:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332776320:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017310.086994:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017310.087000:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017310/real 1734017310] req@ffff8800a73ce680 x1818247332776320/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017310.087010:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332776320/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017310.087017:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017310.087019:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017310.087022:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332776320:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017310.087027:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332776448:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017310.087039:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017310.087045:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017310/real 1734017310] req@ffff8800a73cdc00 x1818247332776448/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017310.087056:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332776448/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017310.087063:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017310.087065:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017310.087068:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332776448:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017310.326457:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017311.328444:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017312.330496:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017313.332517:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017314.334506:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017315.094343:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017315.094353:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017315.094358:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017315.094372:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1577 00000100:00080000:3.0:1734017315.094384:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017315.094394:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017315.094441:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332776576/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017315.094480:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017315.094485:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017315.094487:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017315.094492:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1577 00000100:00080000:3.0:1734017315.094495:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017315.094499:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017315.094511:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017315.094524:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332776704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017315.094535:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017315.094539:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017315.094541:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017315.094545:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1577 00000100:00080000:3.0:1734017315.094547:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017315.094551:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017315.094556:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017315.094565:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332776832/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017315.094574:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017315.094577:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017315.094579:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017315.094582:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1577 00000100:00080000:3.0:1734017315.094585:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017315.094588:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017315.094593:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017315.094601:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332776960/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017315.094611:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017315.094614:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017315.094616:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017315.094619:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332776576:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017315.094620:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1577 00000100:00080000:3.0:1734017315.094623:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017315.094627:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017315.094632:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017315.094642:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332777088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017315.094685:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017315.094689:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017315.094691:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017315.094696:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1577 00000100:00080000:3.0:1734017315.094699:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017315.094703:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017315.094708:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017315.094718:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332777216/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017315.094727:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017315.094730:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017315.094732:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017315.094735:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1577 00000800:00000100:1.0:1734017315.094736:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017315.094739:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017315.094742:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017315.094747:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017315.094756:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332777344/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017315.094761:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017315/real 1734017315] req@ffff8800a73cf100 x1818247332776576/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017315.094782:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332776576/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017315.094794:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017315.094797:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017315.094802:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332776576:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017315.094810:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332776704:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017315.094829:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017315.094836:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017315/real 1734017315] req@ffff8800a73cdc00 x1818247332776704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017315.094848:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332776704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017315.094857:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017315.094860:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017315.094863:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332776704:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017315.094870:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332776832:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017315.094883:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017315.094889:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017315/real 1734017315] req@ffff8800a73ce680 x1818247332776832/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017315.094900:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332776832/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017315.094907:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017315.094910:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017315.094913:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332776832:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017315.094939:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332776960:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017315.094952:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017315.094959:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017315/real 1734017315] req@ffff8800a73cc700 x1818247332776960/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017315.094969:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332776960/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017315.094976:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017315.094979:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017315.094982:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332776960:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017315.094988:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332777088:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017315.095001:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017315.095007:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017315/real 1734017315] req@ffff8800a73cc380 x1818247332777088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017315.095017:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332777088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017315.095025:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017315.095027:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017315.095030:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332777088:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017315.095037:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332777216:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017315.095049:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017315.095055:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017315/real 1734017315] req@ffff8800a73cdf80 x1818247332777216/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017315.095065:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332777216/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017315.095073:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017315.095075:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017315.095078:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332777216:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017315.095084:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332777344:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017315.095096:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017315.095102:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017315/real 1734017315] req@ffff8800a73cf800 x1818247332777344/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017315.095112:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332777344/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017315.095120:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017315.095122:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017315.095125:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332777344:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017315.336489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017316.338453:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017317.340477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017318.342494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017319.344489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017320.102357:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017320.102366:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017320.102370:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017320.102381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1582 00000100:00080000:3.0:1734017320.102391:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017320.102399:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017320.102435:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332777472/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017320.102470:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017320.102474:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017320.102476:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017320.102481:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1582 00000100:00080000:3.0:1734017320.102484:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017320.102488:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017320.102498:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017320.102509:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332777600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017320.102519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017320.102522:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017320.102524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017320.102527:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1582 00000100:00080000:3.0:1734017320.102531:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017320.102534:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017320.102539:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017320.102548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332777728/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017320.102556:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017320.102559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017320.102560:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017320.102564:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1582 00000100:00080000:3.0:1734017320.102599:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017320.102603:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017320.102608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017320.102617:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332777856/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017320.102623:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332777472:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017320.102627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017320.102630:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017320.102632:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017320.102635:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1582 00000100:00080000:3.0:1734017320.102638:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017320.102642:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017320.102646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017320.102654:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332777984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017320.102669:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017320.102672:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017320.102673:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017320.102677:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1582 00000100:00080000:3.0:1734017320.102681:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017320.102684:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017320.102689:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017320.102691:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017320.102696:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332778112/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017320.102704:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017320.102707:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017320.102709:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017320.102712:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017320/real 1734017320] req@ffff8800a73cce00 x1818247332777472/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017320.102713:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1582 00000100:00080000:3.0:1734017320.102715:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017320.102719:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017320.102723:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017320.102732:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332777472/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017320.102732:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332778240/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017320.102743:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017320.102746:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017320.102751:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332777472:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017320.102758:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332777600:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017320.102775:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017320.102781:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017320/real 1734017320] req@ffff8800a73cca80 x1818247332777600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017320.102792:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332777600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017320.102800:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017320.102803:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017320.102806:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332777600:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017320.102812:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332777728:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017320.102825:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017320.102831:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017320/real 1734017320] req@ffff8800a73cf480 x1818247332777728/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017320.102841:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332777728/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017320.102848:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017320.102851:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017320.102854:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332777728:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017320.102878:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332777856:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017320.102891:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017320.102897:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017320/real 1734017320] req@ffff8800a73cfb80 x1818247332777856/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017320.102907:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332777856/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017320.102914:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017320.102916:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017320.102919:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332777856:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017320.102925:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332777984:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017320.102937:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017320.102943:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017320/real 1734017320] req@ffff8800a73cd500 x1818247332777984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017320.102953:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332777984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017320.102961:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017320.102963:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017320.102966:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332777984:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017320.102971:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332778112:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017320.102982:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017320.102988:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017320/real 1734017320] req@ffff8800a73cea00 x1818247332778112/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017320.102997:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332778112/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017320.103004:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017320.103006:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017320.103009:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332778112:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017320.103014:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332778240:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017320.103026:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017320.103032:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017320/real 1734017320] req@ffff8800a73ced80 x1818247332778240/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017320.103042:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332778240/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017320.103050:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017320.103052:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017320.103054:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332778240:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017320.346445:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017321.348490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017322.350503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017323.352484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017324.354482:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017325.110345:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017325.110354:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017325.110359:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017325.110370:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1587 00000100:00080000:3.0:1734017325.110380:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017325.110388:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017325.110424:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332778368/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017325.110459:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017325.110463:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017325.110465:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017325.110470:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1587 00000100:00080000:3.0:1734017325.110473:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017325.110477:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017325.110538:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017325.110553:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332778496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017325.110565:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017325.110568:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017325.110570:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017325.110574:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332778368:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017325.110574:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1587 00000100:00080000:3.0:1734017325.110577:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017325.110581:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017325.110585:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017325.110593:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332778624/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017325.110602:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017325.110606:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017325.110607:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017325.110611:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1587 00000100:00080000:3.0:1734017325.110614:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017325.110617:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017325.110622:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017325.110629:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332778752/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017325.110637:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017325.110640:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1734017325.110641:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017325.110642:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017325.110645:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1587 00000100:00080000:3.0:1734017325.110648:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017325.110651:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017325.110656:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734017325.110664:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017325/real 1734017325] req@ffff8800a73cd880 x1818247332778368/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017325.110664:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332778880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017325.110672:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017325.110675:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017325.110677:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017325.110681:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1587 00000100:00100000:1.0:1734017325.110683:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332778368/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017325.110683:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017325.110687:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017325.110693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734017325.110694:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017325.110697:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017325.110700:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332779008/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017325.110703:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332778368:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017325.110709:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017325.110712:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017325.110713:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017325.110717:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1587 00000100:00080000:3.0:1734017325.110719:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017325.110722:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017325.110725:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332778496:192.168.201.152@tcp:38:kworker.0 00000008:00100000:3.0:1734017325.110726:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017325.110734:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332779136/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017325.110743:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017325.110750:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017325/real 1734017325] req@ffff8800a73ced80 x1818247332778496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017325.110761:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332778496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017325.110771:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017325.110774:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017325.110777:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332778496:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017325.110783:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332778624:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017325.110797:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017325.110803:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017325/real 1734017325] req@ffff8800a73cea00 x1818247332778624/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017325.110813:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332778624/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017325.110820:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017325.110822:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017325.110826:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332778624:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017325.110831:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332778752:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017325.110843:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017325.110849:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017325/real 1734017325] req@ffff8800a73cd500 x1818247332778752/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017325.110858:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332778752/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017325.110866:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017325.110869:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017325.110872:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332778752:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017325.110878:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332778880:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017325.110890:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017325.110896:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017325/real 1734017325] req@ffff8800a73cfb80 x1818247332778880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017325.110906:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332778880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017325.110913:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017325.110916:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017325.110918:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332778880:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017325.110923:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332779008:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017325.110935:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017325.110941:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017325/real 1734017325] req@ffff8800a73cf480 x1818247332779008/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017325.110951:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332779008/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017325.110958:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017325.110961:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017325.110964:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332779008:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017325.110980:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332779136:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017325.110992:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017325.110998:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017325/real 1734017325] req@ffff8800a73cca80 x1818247332779136/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017325.111009:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332779136/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017325.111017:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017325.111019:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017325.111022:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332779136:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017325.356457:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017326.358501:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017327.360506:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017328.362501:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017329.364503:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017330.118339:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017330.118347:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017330.118352:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017330.118362:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1592 00000100:00080000:3.0:1734017330.118372:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017330.118380:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017330.118447:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332779264/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017330.118480:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017330.118487:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017330.118489:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017330.118494:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1592 00000100:00080000:3.0:1734017330.118498:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017330.118501:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017330.118511:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017330.118522:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332779392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017330.118533:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017330.118536:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017330.118538:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017330.118541:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1592 00000100:00080000:3.0:1734017330.118544:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017330.118547:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017330.118552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017330.118560:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332779520/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017330.118568:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017330.118572:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017330.118573:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017330.118577:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1592 00000100:00080000:3.0:1734017330.118579:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017330.118582:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017330.118588:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017330.118595:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332779264:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017330.118596:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332779648/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017330.118605:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017330.118608:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017330.118610:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017330.118613:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1592 00000100:00080000:3.0:1734017330.118616:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017330.118619:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017330.118623:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017330.118631:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332779776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017330.118639:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017330.118642:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017330.118643:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017330.118647:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1592 00000100:00080000:3.0:1734017330.118652:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017330.118655:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017330.118660:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017330.118662:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017330.118667:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332779904/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017330.118676:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017330.118679:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017330.118681:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017330.118684:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017330/real 1734017330] req@ffff8800a73cce00 x1818247332779264/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017330.118685:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1592 00000100:00080000:3.0:1734017330.118688:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017330.118691:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017330.118696:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017330.118703:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332779264/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017330.118703:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332780032/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017330.118714:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017330.118718:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017330.118722:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332779264:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017330.118730:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332779392:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017330.118748:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017330.118754:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017330/real 1734017330] req@ffff8800a73cf800 x1818247332779392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017330.118765:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332779392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017330.118773:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017330.118776:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017330.118779:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332779392:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017330.118785:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332779520:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017330.118798:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017330.118804:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017330/real 1734017330] req@ffff8800a73cdf80 x1818247332779520/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017330.118814:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332779520/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017330.118822:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017330.118824:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017330.118827:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332779520:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017330.118852:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332779648:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017330.118865:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017330.118871:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017330/real 1734017330] req@ffff8800a73cc380 x1818247332779648/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017330.118881:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332779648/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017330.118889:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017330.118891:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017330.118894:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332779648:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017330.118900:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332779776:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017330.118912:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017330.118918:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017330/real 1734017330] req@ffff8800a73cc700 x1818247332779776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017330.118928:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332779776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017330.118935:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017330.118937:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017330.118940:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332779776:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017330.118946:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332779904:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017330.118958:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017330.118964:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017330/real 1734017330] req@ffff8800a73ce680 x1818247332779904/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017330.118973:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332779904/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017330.118980:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017330.118982:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017330.118985:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332779904:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017330.118991:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332780032:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017330.119002:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017330.119008:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017330/real 1734017330] req@ffff8800a73cdc00 x1818247332780032/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017330.119018:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332780032/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017330.119026:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017330.119028:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017330.119031:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332780032:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017330.366469:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017331.368468:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017332.370507:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017333.372503:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017334.374526:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017335.126392:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017335.126401:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017335.126405:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017335.126415:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1597 00000100:00080000:3.0:1734017335.126425:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017335.126433:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017335.126471:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332780160/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017335.126504:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017335.126509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017335.126511:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017335.126515:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1597 00000100:00080000:3.0:1734017335.126519:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017335.126522:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017335.126532:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017335.126544:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332780288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017335.126555:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017335.126559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017335.126561:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017335.126564:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1597 00000100:00080000:3.0:1734017335.126566:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017335.126570:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017335.126574:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017335.126583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332780416/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017335.126591:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017335.126594:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017335.126595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017335.126599:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1597 00000100:00080000:3.0:1734017335.126601:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017335.126605:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017335.126609:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017335.126617:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332780544/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017335.126623:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332780160:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017335.126626:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017335.126629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017335.126631:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017335.126634:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1597 00000100:00080000:3.0:1734017335.126638:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017335.126641:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017335.126646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017335.126654:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332780672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017335.126662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017335.126665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017335.126666:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017335.126670:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1597 00000100:00080000:3.0:1734017335.126673:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017335.126676:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017335.126680:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017335.126687:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332780800/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017335.126690:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017335.126695:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017335.126698:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017335.126700:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017335.126703:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1597 00000100:00080000:3.0:1734017335.126707:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017335.126710:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017335.126712:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017335/real 1734017335] req@ffff8800a73cf100 x1818247332780160/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017335.126715:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017335.126722:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332780928/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017335.126731:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332780160/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017335.126741:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017335.126745:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017335.126749:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332780160:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017335.126756:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332780288:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017335.126773:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017335.126780:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017335/real 1734017335] req@ffff8800a73cdc00 x1818247332780288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017335.126791:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332780288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017335.126799:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017335.126802:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017335.126805:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332780288:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017335.126810:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332780416:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017335.126824:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017335.126829:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017335/real 1734017335] req@ffff8800a73ce680 x1818247332780416/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017335.126839:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332780416/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017335.126847:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017335.126849:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017335.126852:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332780416:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017335.126877:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332780544:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017335.126890:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017335.126896:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017335/real 1734017335] req@ffff8800a73cc700 x1818247332780544/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017335.126906:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332780544/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017335.126913:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017335.126915:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017335.126918:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332780544:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017335.126924:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332780672:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017335.126936:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017335.126941:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017335/real 1734017335] req@ffff8800a73cc380 x1818247332780672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017335.126951:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332780672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017335.126959:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017335.126961:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017335.126964:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332780672:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017335.126969:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332780800:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017335.126980:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017335.126985:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017335/real 1734017335] req@ffff8800a73cdf80 x1818247332780800/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017335.126995:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332780800/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017335.127002:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017335.127004:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017335.127007:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332780800:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017335.127012:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332780928:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017335.127024:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017335.127030:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017335/real 1734017335] req@ffff8800a73cf800 x1818247332780928/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017335.127040:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332780928/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017335.127048:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017335.127050:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017335.127053:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332780928:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017335.376488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017336.378482:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017337.380365:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017338.382438:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017339.384518:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017340.134355:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017340.134364:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017340.134368:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017340.134379:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1602 00000100:00080000:3.0:1734017340.134390:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017340.134398:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017340.134434:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332781056/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017340.134469:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017340.134474:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017340.134476:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017340.134480:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1602 00000100:00080000:3.0:1734017340.134484:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017340.134488:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017340.134497:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017340.134508:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332781184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017340.134518:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017340.134522:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017340.134524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017340.134527:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1602 00000100:00080000:3.0:1734017340.134531:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017340.134535:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017340.134540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017340.134548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332781312/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017340.134557:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017340.134560:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017340.134561:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017340.134565:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1602 00000100:00080000:3.0:1734017340.134567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017340.134570:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017340.134575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017340.134582:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332781440/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017340.134591:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332781056:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017340.134591:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017340.134594:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017340.134596:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017340.134599:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1602 00000100:00080000:3.0:1734017340.134602:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017340.134605:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017340.134609:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017340.134617:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332781568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017340.134632:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017340.134635:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017340.134636:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017340.134640:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1602 00000100:00080000:3.0:1734017340.134652:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017340.134656:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017340.134661:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017340.134668:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332781696/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017340.134676:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017340.134680:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017340.134682:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017340.134683:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332781184:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017340.134685:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1602 00000100:00080000:3.0:1734017340.134687:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017340.134690:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017340.134695:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017340.134703:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332781824/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017340.134704:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332781312:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017340.134728:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332781440:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017340.134744:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332781568:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017340.134758:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332781696:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017340.134772:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332781824:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017340.386484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017341.388492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017342.390454:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1734017343.140484:0:1958:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:2.0:1734017343.140534:0:1958:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1734017343.140553:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734017343.140587:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734017343.140595:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734017343.140602:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734017343.140609:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734017343.140615:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734017343.140622:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734017343.140700:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017340/real 1734017343] req@ffff8800a73cfb80 x1818247332781440/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017343.140729:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332781440/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017343.140749:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017343.140753:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017343.140758:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332781440:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017343.140765:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017340/real 1734017343] req@ffff8800a73cd500 x1818247332781568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017343.140777:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332781568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017343.140785:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017343.140787:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017343.140791:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332781568:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017343.140797:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017340/real 1734017343] req@ffff8800a73cea00 x1818247332781696/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017343.140808:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332781696/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017343.140817:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017343.140821:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017343.140826:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332781696:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017343.140836:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017340/real 1734017343] req@ffff8800a73ced80 x1818247332781824/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017343.140855:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332781824/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017343.140876:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017343.140879:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017343.140885:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332781824:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017343.140895:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017340/real 1734017343] req@ffff8800a73cce00 x1818247332781056/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017343.140914:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332781056/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017343.140926:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017343.140930:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017343.140935:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332781056:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017343.140945:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017340/real 1734017343] req@ffff8800a73cca80 x1818247332781184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017343.140992:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332781184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017343.141005:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017343.141009:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017343.141015:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332781184:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017343.141033:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017340/real 1734017343] req@ffff8800a73cf480 x1818247332781312/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017343.141050:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332781312/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017343.141071:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017343.141076:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017343.141081:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332781312:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017343.392475:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017344.394505:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017345.142383:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017345.142392:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017345.142397:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017345.142407:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1607 00000100:00080000:3.0:1734017345.142418:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017345.142425:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017345.142463:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332781952/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017345.142500:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017345.142506:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017345.142509:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017345.142515:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1607 00000100:00080000:3.0:1734017345.142519:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017345.142525:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017345.142538:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017345.142558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332782080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017345.142574:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017345.142579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017345.142581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017345.142588:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1607 00000100:00080000:3.0:1734017345.142593:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017345.142597:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017345.142604:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:1.0:1734017345.142612:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332781952:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017345.142619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332782208/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017345.142633:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017345.142639:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017345.142641:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017345.142647:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1607 00000100:00080000:3.0:1734017345.142651:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017345.142656:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017345.142662:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017345.142674:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332782336/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017345.142680:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017345.142687:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017345.142692:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017345.142694:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017345.142700:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1607 00000100:00100000:1.0:1734017345.142702:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017345/real 1734017345] req@ffff8800a73cd880 x1818247332781952/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017345.142704:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017345.142710:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017345.142716:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734017345.142721:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332781952/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017345.142728:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332782464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017345.142732:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017345.142735:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017345.142740:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332781952:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017345.142741:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017345.142746:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1734017345.142747:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332782080:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017345.142749:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017345.142754:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1607 00000100:00080000:3.0:1734017345.142758:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017345.142763:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734017345.142765:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017345.142771:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017345/real 1734017345] req@ffff8800a73cf480 x1818247332782080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017345.142771:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017345.142782:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332782080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017345.142783:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332782592/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017345.142790:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017345.142793:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734017345.142795:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017345.142797:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332782080:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017345.142800:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017345.142802:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017345.142807:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1607 00000100:00080000:3.0:1734017345.142812:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017345.142817:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017345.142820:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332782208:192.168.201.152@tcp:8:kworker.0 00000008:00100000:3.0:1734017345.142824:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017345.142834:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017345.142838:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332782720/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017345.142841:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017345/real 1734017345] req@ffff8800a73cca80 x1818247332782208/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017345.142851:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332782208/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017345.142859:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017345.142861:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017345.142864:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332782208:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017345.142869:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332782336:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017345.142881:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017345.142887:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017345/real 1734017345] req@ffff8800a73cce00 x1818247332782336/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017345.142897:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332782336/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017345.142904:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017345.142906:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017345.142909:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332782336:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017345.142914:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332782464:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017345.142926:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017345.142933:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017345/real 1734017345] req@ffff8800a73ced80 x1818247332782464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017345.142943:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332782464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017345.142951:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017345.142953:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017345.142956:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332782464:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017345.142961:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332782592:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017345.142973:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017345.142978:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017345/real 1734017345] req@ffff8800a73cea00 x1818247332782592/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017345.142988:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332782592/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017345.142995:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017345.142997:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017345.143000:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332782592:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017345.143013:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332782720:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017345.143025:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017345.143031:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017345/real 1734017345] req@ffff8800a73cd500 x1818247332782720/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017345.143041:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332782720/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017345.143055:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017345.143058:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017345.143061:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332782720:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017345.396501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017346.398511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017347.400490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017348.402477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017349.404499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017350.150365:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017350.150376:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017350.150382:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017350.150395:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1612 00000100:00080000:3.0:1734017350.150407:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017350.150417:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017350.150466:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332782848/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017350.150505:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017350.150513:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017350.150516:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017350.150521:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1612 00000100:00080000:3.0:1734017350.150524:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017350.150528:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017350.150540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017350.150552:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332782976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017350.150563:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017350.150566:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017350.150568:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017350.150572:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1612 00000100:00080000:3.0:1734017350.150575:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017350.150578:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017350.150583:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017350.150592:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332783104/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017350.150601:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017350.150604:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017350.150606:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017350.150609:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1612 00000100:00080000:3.0:1734017350.150612:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017350.150616:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017350.150621:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017350.150629:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332783232/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017350.150640:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017350.150642:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332782848:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017350.150643:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017350.150645:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017350.150649:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1612 00000100:00080000:3.0:1734017350.150651:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017350.150655:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017350.150660:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017350.150668:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332783360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017350.150676:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017350.150680:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017350.150681:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017350.150685:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1612 00000100:00080000:3.0:1734017350.150688:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017350.150692:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017350.150696:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017350.150704:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332783488/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017350.150712:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017350.150716:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1734017350.150717:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017350.150718:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017350.150723:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1612 00000100:00080000:3.0:1734017350.150726:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017350.150730:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017350.150734:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017350.150742:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017350/real 1734017350] req@ffff8800a73cfb80 x1818247332782848/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017350.150742:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332783616/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017350.150762:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332782848/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017350.150774:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017350.150778:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017350.150783:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332782848:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017350.150791:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332782976:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017350.150809:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017350.150815:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017350/real 1734017350] req@ffff8800a73cf800 x1818247332782976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017350.150826:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332782976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017350.150834:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017350.150837:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017350.150841:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332782976:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017350.150848:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332783104:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017350.150862:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017350.150869:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017350/real 1734017350] req@ffff8800a73cdf80 x1818247332783104/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017350.150879:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332783104/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017350.150887:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017350.150889:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017350.150892:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332783104:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017350.150918:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332783232:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017350.150931:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017350.150937:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017350/real 1734017350] req@ffff8800a73cc380 x1818247332783232/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017350.150947:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332783232/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017350.150955:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017350.150957:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017350.150960:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332783232:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017350.150966:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332783360:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017350.150979:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017350.150985:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017350/real 1734017350] req@ffff8800a73cc700 x1818247332783360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017350.150995:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332783360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017350.151003:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017350.151005:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017350.151008:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332783360:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017350.151014:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332783488:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017350.151026:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017350.151032:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017350/real 1734017350] req@ffff8800a73ce680 x1818247332783488/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017350.151042:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332783488/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017350.151049:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017350.151051:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017350.151102:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332783488:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017350.151108:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332783616:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017350.151126:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017350.151133:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017350/real 1734017350] req@ffff8800a73cdc00 x1818247332783616/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017350.151144:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332783616/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017350.151159:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017350.151162:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017350.151165:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332783616:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017350.406480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017351.408483:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017352.410494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017353.412449:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017354.414491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017355.158395:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017355.158406:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017355.158411:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017355.158425:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1617 00000100:00080000:3.0:1734017355.158437:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017355.158447:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017355.158495:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332783744/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017355.158536:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017355.158542:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017355.158544:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017355.158549:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1617 00000100:00080000:3.0:1734017355.158553:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017355.158557:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017355.158570:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017355.158586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332783872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017355.158598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017355.158601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017355.158603:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017355.158607:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1617 00000100:00080000:3.0:1734017355.158609:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017355.158613:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017355.158618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017355.158627:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332784000/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017355.158636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017355.158640:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017355.158641:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017355.158645:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1617 00000100:00080000:3.0:1734017355.158647:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017355.158651:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017355.158656:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017355.158661:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332783744:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017355.158665:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332784128/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017355.158675:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017355.158679:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017355.158680:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017355.158683:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1617 00000100:00080000:3.0:1734017355.158688:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017355.158692:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017355.158697:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017355.158706:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332784256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017355.158714:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017355.158718:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017355.158719:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017355.158723:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1617 00000100:00080000:3.0:1734017355.158727:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017355.158730:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017355.158734:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017355.158735:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017355.158742:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332784384/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017355.158751:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017355.158754:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017355.158756:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017355.158759:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1617 00000100:00100000:1.0:1734017355.158760:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017355/real 1734017355] req@ffff8800a73cf100 x1818247332783744/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017355.158762:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017355.158766:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017355.158770:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017355.158778:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332784512/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017355.158780:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332783744/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017355.158792:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017355.158796:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017355.158801:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332783744:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017355.158810:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332783872:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017355.158828:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017355.158835:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017355/real 1734017355] req@ffff8800a73cdc00 x1818247332783872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017355.158847:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332783872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017355.158854:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017355.158858:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017355.158862:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332783872:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017355.158869:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332784000:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017355.158883:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017355.158889:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017355/real 1734017355] req@ffff8800a73ce680 x1818247332784000/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017355.158899:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332784000/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017355.158907:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017355.158910:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017355.158913:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332784000:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017355.158941:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332784128:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017355.158953:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017355.158960:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017355/real 1734017355] req@ffff8800a73cc700 x1818247332784128/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017355.159017:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332784128/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017355.159026:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017355.159028:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017355.159032:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332784128:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017355.159038:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332784256:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017355.159058:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017355.159065:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017355/real 1734017355] req@ffff8800a73cc380 x1818247332784256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017355.159076:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332784256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017355.159084:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017355.159086:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017355.159089:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332784256:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017355.159095:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332784384:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017355.159107:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017355.159113:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017355/real 1734017355] req@ffff8800a73cdf80 x1818247332784384/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017355.159123:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332784384/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017355.159130:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017355.159133:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017355.159136:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332784384:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017355.159141:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332784512:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017355.159153:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017355.159159:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017355/real 1734017355] req@ffff8800a73cf800 x1818247332784512/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017355.159176:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332784512/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017355.159183:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017355.159185:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017355.159188:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332784512:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017355.416491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017356.418449:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017357.420496:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017358.422454:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017359.424531:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017360.166403:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017360.166414:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017360.166420:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017360.166434:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1622 00000100:00080000:3.0:1734017360.166446:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017360.166456:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017360.166506:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332784640/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017360.166546:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017360.166550:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017360.166552:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017360.166558:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1622 00000100:00080000:3.0:1734017360.166562:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017360.166566:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017360.166578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017360.166590:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332784768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017360.166600:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017360.166603:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017360.166605:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017360.166610:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1622 00000100:00080000:3.0:1734017360.166614:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017360.166618:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017360.166623:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017360.166632:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332784896/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017360.166641:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017360.166644:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017360.166645:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017360.166649:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1622 00000100:00080000:3.0:1734017360.166652:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017360.166655:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017360.166660:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017360.166669:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332785024/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017360.166673:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332784640:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017360.166679:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017360.166682:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017360.166684:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017360.166688:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1622 00000100:00080000:3.0:1734017360.166690:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017360.166694:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017360.166698:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017360.166707:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332785152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017360.166723:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017360.166726:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017360.166727:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017360.166732:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1622 00000100:00080000:3.0:1734017360.166745:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017360.166748:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017360.166749:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017360.166754:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017360.166763:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332785280/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017360.166772:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017360.166773:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017360/real 1734017360] req@ffff8800a73cfb80 x1818247332784640/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017360.166775:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017360.166777:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017360.166781:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1622 00000100:00080000:3.0:1734017360.166784:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017360.166787:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017360.166792:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017360.166793:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332784640/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017360.166804:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332785408/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017360.166805:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017360.166809:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017360.166814:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332784640:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017360.166823:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332784768:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017360.166842:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017360.166849:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017360/real 1734017360] req@ffff8800a73cd500 x1818247332784768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017360.166860:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332784768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017360.166869:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017360.166872:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017360.166875:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332784768:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017360.166882:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332784896:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017360.166951:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017360.166959:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017360/real 1734017360] req@ffff8800a73cea00 x1818247332784896/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017360.166970:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332784896/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017360.166978:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017360.166980:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017360.166983:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332784896:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017360.167010:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332785024:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017360.167023:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017360.167030:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017360/real 1734017360] req@ffff8800a73ced80 x1818247332785024/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017360.167040:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332785024/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017360.167047:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017360.167050:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017360.167053:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332785024:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017360.167060:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332785152:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017360.167073:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017360.167079:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017360/real 1734017360] req@ffff8800a73cce00 x1818247332785152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017360.167090:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332785152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017360.167098:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017360.167100:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017360.167103:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332785152:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017360.167109:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332785280:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017360.167121:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017360.167127:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017360/real 1734017360] req@ffff8800a73cca80 x1818247332785280/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017360.167137:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332785280/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017360.167144:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017360.167146:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017360.167149:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332785280:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017360.167154:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332785408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017360.167167:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017360.167173:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017360/real 1734017360] req@ffff8800a73cf480 x1818247332785408/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017360.167184:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332785408/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017360.167191:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017360.167193:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017360.167196:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332785408:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017360.426453:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017361.428495:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017362.430480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017363.432521:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017364.434487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017365.174375:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017365.174385:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017365.174390:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017365.174404:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1627 00000100:00080000:3.0:1734017365.174417:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017365.174427:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017365.174476:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332785536/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017365.174515:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017365.174519:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017365.174522:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017365.174527:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1627 00000100:00080000:3.0:1734017365.174530:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017365.174535:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017365.174547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017365.174562:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332785664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017365.174572:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017365.174576:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017365.174578:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017365.174582:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1627 00000100:00080000:3.0:1734017365.174585:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017365.174589:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017365.174594:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017365.174603:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332785792/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017365.174612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017365.174616:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017365.174618:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017365.174621:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1627 00000100:00080000:3.0:1734017365.174624:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017365.174628:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017365.174633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017365.174641:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332785536:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017365.174642:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332785920/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017365.174651:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017365.174655:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017365.174656:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017365.174660:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1627 00000100:00080000:3.0:1734017365.174663:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017365.174666:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017365.174671:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017365.174679:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332786048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017365.174688:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017365.174691:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017365.174692:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017365.174697:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1627 00000100:00080000:3.0:1734017365.174710:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017365.174714:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017365.174715:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017365.174721:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017365.174730:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332786176/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017365.174739:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017365.174740:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017365/real 1734017365] req@ffff8800a73cd880 x1818247332785536/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017365.174743:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017365.174744:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017365.174748:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1627 00000100:00080000:3.0:1734017365.174751:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017365.174754:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017365.174759:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017365.174760:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332785536/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017365.174768:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332786304/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017365.174773:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017365.174776:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017365.174782:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332785536:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017365.174791:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332785664:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017365.174857:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017365.174866:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017365/real 1734017365] req@ffff8800a73cf480 x1818247332785664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017365.174878:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332785664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017365.174887:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017365.174891:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017365.174894:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332785664:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017365.174901:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332785792:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017365.174916:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017365.174922:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017365/real 1734017365] req@ffff8800a73cca80 x1818247332785792/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017365.174932:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332785792/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017365.174940:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017365.174942:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017365.174945:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332785792:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017365.174971:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332785920:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017365.174984:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017365.174991:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017365/real 1734017365] req@ffff8800a73cce00 x1818247332785920/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017365.175001:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332785920/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017365.175009:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017365.175011:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017365.175015:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332785920:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017365.175022:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332786048:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017365.175034:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017365.175041:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017365/real 1734017365] req@ffff8800a73ced80 x1818247332786048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017365.175052:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332786048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017365.175060:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017365.175062:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017365.175065:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332786048:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017365.175071:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332786176:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017365.175083:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017365.175089:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017365/real 1734017365] req@ffff8800a73cea00 x1818247332786176/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017365.175099:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332786176/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017365.175106:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017365.175108:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017365.175111:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332786176:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017365.175117:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332786304:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017365.175130:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017365.175136:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017365/real 1734017365] req@ffff8800a73cd500 x1818247332786304/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017365.175147:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332786304/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017365.175154:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017365.175157:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017365.175160:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332786304:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017365.436459:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017366.438507:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017367.440500:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017368.442496:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017369.444493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017370.182390:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017370.182399:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017370.182405:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017370.182418:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1632 00000100:00080000:3.0:1734017370.182430:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017370.182440:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017370.182487:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332786432/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017370.182526:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017370.182534:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017370.182537:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017370.182542:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1632 00000100:00080000:3.0:1734017370.182545:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017370.182550:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017370.182561:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017370.182573:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332786560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017370.182583:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017370.182586:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017370.182588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017370.182592:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1632 00000100:00080000:3.0:1734017370.182595:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017370.182599:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017370.182604:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017370.182613:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332786688/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017370.182621:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017370.182624:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017370.182626:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017370.182630:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1632 00000100:00080000:3.0:1734017370.182632:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017370.182636:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017370.182641:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017370.182651:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332786816/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017370.182655:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332786432:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017370.182661:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017370.182665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017370.182666:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017370.182670:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1632 00000100:00080000:3.0:1734017370.182673:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017370.182676:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017370.182681:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017370.182689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332786944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017370.182697:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017370.182700:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017370.182702:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017370.182707:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1632 00000100:00080000:3.0:1734017370.182710:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017370.182751:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017370.182757:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017370.182767:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332787072/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017370.182773:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017370.182776:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017370.182780:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017370.182783:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017370.182787:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1632 00000100:00080000:3.0:1734017370.182792:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017370.182796:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017370.182798:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017370/real 1734017370] req@ffff8800a73cfb80 x1818247332786432/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017370.182800:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017370.182812:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332787200/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017370.182819:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332786432/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017370.182831:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017370.182835:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017370.182840:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332786432:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017370.182848:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332786560:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017370.182869:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017370.182876:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017370/real 1734017370] req@ffff8800a73cf800 x1818247332786560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017370.182887:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332786560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017370.182896:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017370.182899:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017370.182902:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332786560:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017370.182909:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332786688:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017370.182923:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017370.182929:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017370/real 1734017370] req@ffff8800a73cdf80 x1818247332786688/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017370.182939:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332786688/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017370.182946:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017370.182948:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017370.182952:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332786688:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017370.182978:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332786816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017370.182990:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017370.182997:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017370/real 1734017370] req@ffff8800a73cc380 x1818247332786816/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017370.183007:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332786816/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017370.183015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017370.183017:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017370.183020:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332786816:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017370.183027:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332786944:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017370.183040:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017370.183046:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017370/real 1734017370] req@ffff8800a73cc700 x1818247332786944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017370.183056:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332786944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017370.183063:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017370.183065:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017370.183068:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332786944:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017370.183075:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332787072:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017370.183087:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017370.183093:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017370/real 1734017370] req@ffff8800a73ce680 x1818247332787072/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017370.183103:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332787072/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017370.183110:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017370.183113:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017370.183116:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332787072:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017370.183122:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332787200:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017370.183134:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017370.183140:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017370/real 1734017370] req@ffff8800a73cdc00 x1818247332787200/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017370.183151:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332787200/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017370.183158:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017370.183161:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017370.183164:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332787200:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017370.446483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017371.448497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017372.450417:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017373.452507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017374.454484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017375.190362:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017375.190370:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017375.190375:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017375.190386:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1637 00000100:00080000:3.0:1734017375.190396:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017375.190404:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017375.190441:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332787328/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017375.190476:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017375.190481:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017375.190483:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017375.190487:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1637 00000100:00080000:3.0:1734017375.190491:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017375.190494:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017375.190504:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017375.190516:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332787456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017375.190528:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017375.190531:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017375.190533:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017375.190536:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1637 00000100:00080000:3.0:1734017375.190539:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017375.190542:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017375.190547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017375.190555:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332787584/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017375.190564:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017375.190567:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017375.190568:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017375.190572:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1637 00000100:00080000:3.0:1734017375.190574:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017375.190577:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017375.190582:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017375.190586:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332787328:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017375.190589:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332787712/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017375.190598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017375.190601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017375.190603:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017375.190606:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1637 00000100:00080000:3.0:1734017375.190610:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017375.190613:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017375.190617:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017375.190624:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332787840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017375.190662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017375.190665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017375.190667:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017375.190671:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1637 00000100:00080000:3.0:1734017375.190682:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017375.190686:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734017375.190689:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017375.190690:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017375.190699:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332787968/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017375.190708:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017375.190711:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734017375.190712:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017375/real 1734017375] req@ffff8800a73cf100 x1818247332787328/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017375.190713:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017375.190716:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1637 00000100:00080000:3.0:1734017375.190719:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017375.190722:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017375.190726:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017375.190732:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332787328/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017375.190735:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332788096/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017375.190743:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017375.190746:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017375.190751:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332787328:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017375.190758:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332787456:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017375.190777:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017375.190784:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017375/real 1734017375] req@ffff8800a73cdc00 x1818247332787456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017375.190795:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332787456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017375.190804:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017375.190806:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017375.190810:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332787456:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017375.190816:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332787584:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017375.190828:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017375.190834:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017375/real 1734017375] req@ffff8800a73ce680 x1818247332787584/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017375.190844:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332787584/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017375.190852:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017375.190854:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017375.190857:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332787584:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017375.190881:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332787712:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017375.190894:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017375.190900:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017375/real 1734017375] req@ffff8800a73cc700 x1818247332787712/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017375.190910:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332787712/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017375.190918:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017375.190920:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017375.190923:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332787712:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017375.190929:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332787840:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017375.190941:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017375.190947:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017375/real 1734017375] req@ffff8800a73cc380 x1818247332787840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017375.190957:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332787840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017375.190965:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017375.190967:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017375.190970:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332787840:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017375.190975:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332787968:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017375.190986:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017375.190992:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017375/real 1734017375] req@ffff8800a73cdf80 x1818247332787968/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017375.191002:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332787968/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017375.191009:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017375.191011:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017375.191014:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332787968:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017375.191019:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332788096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017375.191031:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017375.191037:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017375/real 1734017375] req@ffff8800a73cf800 x1818247332788096/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017375.191048:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332788096/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017375.191056:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017375.191058:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017375.191061:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332788096:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017375.456466:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017376.458480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017377.460480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017378.462484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017379.464490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017380.198356:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017380.198365:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017380.198369:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017380.198380:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1642 00000100:00080000:3.0:1734017380.198389:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017380.198397:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017380.198432:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332788224/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017380.198467:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017380.198472:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017380.198474:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017380.198478:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1642 00000100:00080000:3.0:1734017380.198481:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017380.198485:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017380.198494:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017380.198505:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332788352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017380.198515:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017380.198518:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017380.198520:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017380.198524:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1642 00000100:00080000:3.0:1734017380.198527:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017380.198531:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017380.198536:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017380.198580:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332788480/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017380.198590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017380.198593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017380.198595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017380.198598:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1642 00000100:00080000:3.0:1734017380.198601:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017380.198604:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017380.198609:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017380.198617:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332788608/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017380.198626:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017380.198627:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332788224:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017380.198630:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017380.198632:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017380.198636:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1642 00000100:00080000:3.0:1734017380.198639:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017380.198642:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017380.198646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017380.198654:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332788736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017380.198670:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017380.198673:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017380.198675:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017380.198679:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1642 00000100:00080000:3.0:1734017380.198683:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017380.198686:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017380.198691:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017380.198694:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017380.198698:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332788864/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017380.198706:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017380.198709:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017380.198711:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017380.198714:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1642 00000100:00100000:1.0:1734017380.198717:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017380/real 1734017380] req@ffff8800a73cfb80 x1818247332788224/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017380.198717:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017380.198721:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017380.198725:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017380.198734:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332788992/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017380.198736:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332788224/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017380.198747:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017380.198750:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017380.198755:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332788224:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017380.198762:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332788352:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017380.198779:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017380.198786:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017380/real 1734017380] req@ffff8800a73cd500 x1818247332788352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017380.198797:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332788352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017380.198805:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017380.198808:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017380.198811:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332788352:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017380.198817:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332788480:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017380.198830:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017380.198836:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017380/real 1734017380] req@ffff8800a73cea00 x1818247332788480/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017380.198846:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332788480/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017380.198853:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017380.198855:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017380.198859:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332788480:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017380.198883:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332788608:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017380.198895:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017380.198901:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017380/real 1734017380] req@ffff8800a73ced80 x1818247332788608/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017380.198911:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332788608/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017380.198919:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017380.198921:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017380.198924:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332788608:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017380.198929:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332788736:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017380.198941:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017380.198948:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017380/real 1734017380] req@ffff8800a73cce00 x1818247332788736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017380.198958:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332788736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017380.198966:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017380.198968:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017380.198971:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332788736:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017380.198977:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332788864:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017380.198988:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017380.198994:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017380/real 1734017380] req@ffff8800a73cca80 x1818247332788864/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017380.199003:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332788864/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017380.199010:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017380.199012:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017380.199015:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332788864:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017380.199020:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332788992:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017380.199032:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017380.199038:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017380/real 1734017380] req@ffff8800a73cf480 x1818247332788992/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017380.199049:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332788992/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017380.199057:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017380.199059:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017380.199062:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332788992:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017380.466479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017381.468474:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017382.470469:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017383.472478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017384.474546:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017385.206354:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017385.206362:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017385.206367:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017385.206378:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1647 00000100:00080000:3.0:1734017385.206388:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017385.206396:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017385.206432:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332789120/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017385.206504:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017385.206509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017385.206511:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017385.206516:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1647 00000100:00080000:3.0:1734017385.206520:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017385.206524:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017385.206534:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017385.206547:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332789248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017385.206557:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017385.206561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017385.206562:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017385.206566:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1647 00000100:00080000:3.0:1734017385.206568:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017385.206572:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017385.206576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:1.0:1734017385.206579:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332789120:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017385.206585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332789376/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017385.206594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017385.206598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017385.206599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017385.206603:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1647 00000100:00080000:3.0:1734017385.206605:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017385.206608:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017385.206613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017385.206620:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332789504/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017385.206629:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017385.206632:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017385.206633:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017385.206637:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1647 00000100:00080000:3.0:1734017385.206639:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017385.206643:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734017385.206644:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017385.206647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017385.206654:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332789632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017385.206662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017385.206666:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1734017385.206667:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017385/real 1734017385] req@ffff8800a73cd880 x1818247332789120/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017385.206667:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017385.206671:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1647 00000100:00080000:3.0:1734017385.206674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017385.206677:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017385.206682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017385.206686:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332789120/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017385.206690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332789760/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017385.206697:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017385.206698:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1734017385.206700:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734017385.206701:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017385.206703:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017385.206705:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332789120:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017385.206706:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1647 00000100:00080000:3.0:1734017385.206709:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017385.206712:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332789248:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017385.206712:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017385.206717:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017385.206724:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332789888/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017385.206730:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017385.206737:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017385/real 1734017385] req@ffff8800a73cf480 x1818247332789248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017385.206748:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332789248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017385.206757:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017385.206759:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017385.206763:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332789248:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017385.206786:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332789376:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017385.206800:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017385.206806:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017385/real 1734017385] req@ffff8800a73cca80 x1818247332789376/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017385.206816:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332789376/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017385.206824:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017385.206826:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017385.206829:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332789376:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017385.206835:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332789504:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017385.206846:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017385.206852:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017385/real 1734017385] req@ffff8800a73cce00 x1818247332789504/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017385.206862:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332789504/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017385.206869:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017385.206871:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017385.206874:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332789504:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017385.206880:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332789632:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017385.206893:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017385.206899:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017385/real 1734017385] req@ffff8800a73ced80 x1818247332789632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017385.206909:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332789632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017385.206916:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017385.206918:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017385.206921:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332789632:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017385.206927:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332789760:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017385.206939:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017385.206945:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017385/real 1734017385] req@ffff8800a73cea00 x1818247332789760/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017385.206955:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332789760/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017385.206962:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017385.206965:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017385.206968:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332789760:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017385.206973:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332789888:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017385.206985:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017385.206990:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017385/real 1734017385] req@ffff8800a73cd500 x1818247332789888/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017385.207001:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332789888/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017385.207009:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017385.207011:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017385.207014:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332789888:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017385.476503:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017386.478510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017387.480495:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017388.482483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017389.484466:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017390.214352:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017390.214361:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017390.214365:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017390.214419:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1652 00000100:00080000:3.0:1734017390.214433:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017390.214442:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017390.214477:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332790016/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017390.214510:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017390.214517:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017390.214519:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017390.214524:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1652 00000100:00080000:3.0:1734017390.214527:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017390.214531:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017390.214540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017390.214551:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332790144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017390.214561:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017390.214564:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017390.214566:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017390.214570:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1652 00000100:00080000:3.0:1734017390.214572:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017390.214576:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017390.214581:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017390.214588:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332790272/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017390.214597:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017390.214600:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017390.214602:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017390.214605:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1652 00000100:00080000:3.0:1734017390.214607:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017390.214610:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017390.214616:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017390.214623:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332790400/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017390.214629:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332790016:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017390.214633:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017390.214636:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017390.214637:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017390.214641:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1652 00000100:00080000:3.0:1734017390.214643:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017390.214647:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017390.214651:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017390.214659:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332790528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017390.214667:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017390.214670:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017390.214671:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017390.214675:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1652 00000100:00080000:3.0:1734017390.214678:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017390.214681:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017390.214685:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017390.214692:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332790656/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017390.214695:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017390.214700:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017390.214704:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017390.214706:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017390.214710:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1652 00000100:00080000:3.0:1734017390.214713:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017390.214716:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017390.214717:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017390/real 1734017390] req@ffff8800a73cfb80 x1818247332790016/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017390.214720:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017390.214729:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332790784/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017390.214736:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332790016/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017390.214747:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017390.214750:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017390.214755:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332790016:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017390.214762:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332790144:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017390.214778:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017390.214785:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017390/real 1734017390] req@ffff8800a73cf800 x1818247332790144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017390.214796:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332790144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017390.214804:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017390.214807:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017390.214810:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332790144:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017390.214816:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332790272:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017390.214829:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017390.214835:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017390/real 1734017390] req@ffff8800a73cdf80 x1818247332790272/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017390.214845:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332790272/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017390.214853:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017390.214855:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017390.214858:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332790272:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017390.214882:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332790400:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017390.214894:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017390.214901:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017390/real 1734017390] req@ffff8800a73cc380 x1818247332790400/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017390.214911:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332790400/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017390.214919:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017390.214921:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017390.214924:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332790400:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017390.214930:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332790528:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017390.214941:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017390.214947:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017390/real 1734017390] req@ffff8800a73cc700 x1818247332790528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017390.214957:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332790528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017390.214965:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017390.214967:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017390.214970:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332790528:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017390.214975:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332790656:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017390.214986:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017390.214992:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017390/real 1734017390] req@ffff8800a73ce680 x1818247332790656/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017390.215002:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332790656/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017390.215009:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017390.215011:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017390.215014:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332790656:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017390.215019:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332790784:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017390.215032:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017390.215038:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017390/real 1734017390] req@ffff8800a73cdc00 x1818247332790784/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017390.215048:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332790784/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017390.215055:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017390.215057:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017390.215060:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332790784:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017390.486468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017391.488476:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017392.490489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017393.492390:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017394.494490:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017395.222377:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017395.222386:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017395.222390:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017395.222401:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1657 00000100:00080000:3.0:1734017395.222411:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017395.222419:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017395.222456:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332790912/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017395.222490:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017395.222494:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017395.222496:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017395.222500:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1657 00000100:00080000:3.0:1734017395.222504:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017395.222507:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017395.222517:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017395.222529:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332791040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017395.222540:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017395.222544:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017395.222545:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017395.222549:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1657 00000100:00080000:3.0:1734017395.222551:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017395.222555:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017395.222559:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017395.222567:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332791168/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017395.222576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017395.222579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017395.222580:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017395.222584:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1657 00000100:00080000:3.0:1734017395.222586:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017395.222589:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017395.222593:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017395.222600:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332790912:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017395.222601:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332791296/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017395.222610:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017395.222613:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017395.222615:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017395.222618:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1657 00000100:00080000:3.0:1734017395.222621:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017395.222625:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017395.222629:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017395.222637:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332791424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017395.222645:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017395.222647:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017395.222649:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017395.222653:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1657 00000100:00080000:3.0:1734017395.222656:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017395.222659:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017395.222663:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017395.222665:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017395.222671:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332791552/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017395.222679:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017395.222682:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017395.222683:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017395.222687:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1657 00000100:00100000:1.0:1734017395.222688:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017395/real 1734017395] req@ffff8800a73cf100 x1818247332790912/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017395.222689:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017395.222693:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017395.222697:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017395.222705:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332791680/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017395.222707:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332790912/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017395.222717:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017395.222720:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017395.222725:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332790912:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017395.222732:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332791040:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017395.222749:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017395.222756:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017395/real 1734017395] req@ffff8800a73cdc00 x1818247332791040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017395.222767:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332791040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017395.222776:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017395.222778:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017395.222782:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332791040:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017395.222788:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332791168:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017395.222801:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017395.222807:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017395/real 1734017395] req@ffff8800a73ce680 x1818247332791168/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017395.222817:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332791168/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017395.222824:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017395.222827:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017395.222830:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332791168:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017395.222855:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332791296:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017395.222867:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017395.222873:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017395/real 1734017395] req@ffff8800a73cc700 x1818247332791296/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017395.222884:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332791296/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017395.222892:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017395.222894:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017395.222897:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332791296:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017395.222902:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332791424:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017395.222914:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017395.222920:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017395/real 1734017395] req@ffff8800a73cc380 x1818247332791424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017395.222930:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332791424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017395.222937:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017395.222939:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017395.222943:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332791424:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017395.222948:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332791552:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017395.222959:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017395.222965:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017395/real 1734017395] req@ffff8800a73cdf80 x1818247332791552/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017395.222975:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332791552/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017395.222982:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017395.222985:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017395.222988:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332791552:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017395.222993:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332791680:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017395.223004:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017395.223010:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017395/real 1734017395] req@ffff8800a73cf800 x1818247332791680/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017395.223021:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332791680/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017395.223028:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017395.223030:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017395.223033:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332791680:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017395.496489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017396.498496:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017397.500475:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017398.502468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017399.504462:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017400.230332:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017400.230341:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017400.230346:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017400.230357:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1662 00000100:00080000:3.0:1734017400.230366:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017400.230374:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017400.230411:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332791808/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017400.230446:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017400.230450:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017400.230452:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017400.230456:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1662 00000100:00080000:3.0:1734017400.230460:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017400.230463:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017400.230473:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017400.230483:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332791936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017400.230493:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017400.230497:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017400.230498:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017400.230502:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1662 00000100:00080000:3.0:1734017400.230506:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017400.230509:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017400.230514:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017400.230522:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332792064/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017400.230531:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017400.230534:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017400.230535:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017400.230539:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1662 00000100:00080000:3.0:1734017400.230541:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017400.230544:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017400.230549:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017400.230556:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332792192/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017400.230565:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017400.230566:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332791808:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017400.230568:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017400.230569:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017400.230573:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1662 00000100:00080000:3.0:1734017400.230576:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017400.230579:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017400.230584:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017400.230591:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332792320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017400.230606:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017400.230609:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017400.230611:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017400.230614:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1662 00000100:00080000:3.0:1734017400.230617:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017400.230620:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017400.230624:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017400.230632:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332792448/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017400.230634:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017400.230640:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017400.230643:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017400.230644:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017400.230648:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1662 00000100:00080000:3.0:1734017400.230651:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017400.230655:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017400.230656:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017400/real 1734017400] req@ffff8800a73cfb80 x1818247332791808/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017400.230659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017400.230667:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332792576/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017400.230675:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332791808/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017400.230686:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017400.230689:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017400.230694:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332791808:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017400.230701:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332791936:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017400.230718:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017400.230725:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017400/real 1734017400] req@ffff8800a73cd500 x1818247332791936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017400.230736:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332791936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017400.230745:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017400.230748:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017400.230751:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332791936:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017400.230757:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332792064:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017400.230770:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017400.230776:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017400/real 1734017400] req@ffff8800a73cea00 x1818247332792064/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017400.230786:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332792064/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017400.230793:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017400.230796:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017400.230799:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332792064:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017400.230823:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332792192:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017400.230836:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017400.230842:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017400/real 1734017400] req@ffff8800a73ced80 x1818247332792192/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017400.230853:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332792192/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017400.230860:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017400.230862:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017400.230865:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332792192:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017400.230870:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332792320:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017400.230882:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017400.230888:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017400/real 1734017400] req@ffff8800a73cce00 x1818247332792320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017400.230898:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332792320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017400.230906:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017400.230908:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017400.230911:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332792320:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017400.230916:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332792448:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017400.230927:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017400.230933:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017400/real 1734017400] req@ffff8800a73cca80 x1818247332792448/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017400.230943:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332792448/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017400.230949:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017400.230951:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017400.230954:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332792448:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017400.230959:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332792576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017400.230971:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017400.230977:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017400/real 1734017400] req@ffff8800a73cf480 x1818247332792576/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017400.230987:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332792576/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017400.230995:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017400.230997:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017400.231000:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332792576:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017400.506484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017401.508477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017402.510477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017403.512405:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017404.514476:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017405.238363:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017405.238372:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017405.238376:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017405.238388:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1667 00000100:00080000:3.0:1734017405.238398:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017405.238406:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017405.238442:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332792704/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017405.238478:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017405.238482:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017405.238484:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017405.238489:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1667 00000100:00080000:3.0:1734017405.238492:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017405.238496:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017405.238505:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017405.238518:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332792832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017405.238528:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017405.238531:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017405.238533:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017405.238537:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1667 00000100:00080000:3.0:1734017405.238540:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017405.238543:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017405.238548:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017405.238556:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332792960/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017405.238564:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017405.238568:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017405.238570:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017405.238573:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1667 00000100:00080000:3.0:1734017405.238576:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017405.238579:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017405.238583:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017405.238589:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332792704:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017405.238591:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332793088/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017405.238600:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017405.238603:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017405.238604:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017405.238608:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1667 00000100:00080000:3.0:1734017405.238610:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017405.238614:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017405.238618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017405.238625:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332793216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017405.238633:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017405.238636:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017405.238637:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017405.238641:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1667 00000100:00080000:3.0:1734017405.238644:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017405.238647:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017405.238653:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017405.238660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332793344/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017405.238671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017405.238675:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017405.238677:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017405.238680:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332792832:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017405.238683:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1667 00000100:00080000:3.0:1734017405.238688:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017405.238693:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017405.238699:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017405.238701:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332792960:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734017405.238713:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332793472/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017405.238723:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332793088:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017405.238737:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332793216:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017405.238752:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332793344:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017405.238768:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332793472:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017405.516377:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017406.518501:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017407.520479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734017408.244544:0:1956:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734017408.244602:0:1956:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734017408.244623:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017408.244659:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017408.244667:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017408.244674:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017408.244680:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017408.244686:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017408.244692:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734017408.244781:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017405/real 1734017408] req@ffff8800a73cd500 x1818247332793472/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017408.244823:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332793472/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017408.244835:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017408.244838:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017408.244842:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332793472:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017408.244847:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017405/real 1734017408] req@ffff8800a73cce00 x1818247332793088/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017408.244853:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332793088/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017408.244858:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017408.244860:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017408.244862:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332793088:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017408.244867:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017405/real 1734017408] req@ffff8800a73ced80 x1818247332793216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017408.244878:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332793216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017408.244885:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017408.244887:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017408.244890:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332793216:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017408.244896:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017405/real 1734017408] req@ffff8800a73cea00 x1818247332793344/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017408.244906:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332793344/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017408.244926:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017408.244928:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017408.244931:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332793344:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017408.244937:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017405/real 1734017408] req@ffff8800a73cd880 x1818247332792704/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017408.244947:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332792704/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017408.244954:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017408.244956:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017408.244959:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332792704:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017408.244964:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017405/real 1734017408] req@ffff8800a73cf480 x1818247332792832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017408.244974:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332792832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017408.244980:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017408.244982:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017408.245003:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332792832:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017408.245013:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017405/real 1734017408] req@ffff8800a73cca80 x1818247332792960/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017408.245041:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332792960/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017408.245061:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017408.245136:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017408.245144:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332792960:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017408.522491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017409.524434:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017410.246353:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017410.246362:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017410.246366:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017410.246377:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1672 00000100:00080000:3.0:1734017410.246386:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017410.246394:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017410.246429:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332793600/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017410.246464:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017410.246471:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017410.246473:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017410.246478:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1672 00000100:00080000:3.0:1734017410.246481:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017410.246485:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017410.246495:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017410.246505:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332793728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017410.246515:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017410.246518:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017410.246520:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017410.246523:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1672 00000100:00080000:3.0:1734017410.246526:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017410.246530:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017410.246534:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017410.246542:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332793856/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017410.246551:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017410.246554:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017410.246555:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017410.246558:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1672 00000100:00080000:3.0:1734017410.246561:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017410.246564:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017410.246569:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017410.246574:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332793600:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017410.246577:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332793984/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017410.246587:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017410.246590:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017410.246591:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017410.246595:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1672 00000100:00080000:3.0:1734017410.246597:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017410.246600:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017410.246604:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017410.246612:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332794112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017410.246620:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017410.246623:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017410.246624:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017410.246628:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1672 00000100:00080000:3.0:1734017410.246631:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017410.246634:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734017410.246638:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017410.246638:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017410.246646:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332794240/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017410.246654:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017410.246657:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734017410.246659:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017410/real 1734017410] req@ffff8800a73cfb80 x1818247332793600/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017410.246659:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017410.246663:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1672 00000100:00080000:3.0:1734017410.246666:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017410.246669:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017410.246673:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017410.246679:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332793600/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017410.246683:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332794368/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017410.246690:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017410.246693:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017410.246698:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332793600:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017410.246705:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332793728:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017410.246722:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017410.246728:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017410/real 1734017410] req@ffff8800a73cf800 x1818247332793728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017410.246740:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332793728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017410.246747:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017410.246750:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017410.246754:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332793728:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017410.246760:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332793856:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017410.246773:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017410.246779:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017410/real 1734017410] req@ffff8800a73cdf80 x1818247332793856/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017410.246789:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332793856/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017410.246797:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017410.246799:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017410.246803:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332793856:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017410.246826:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332793984:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017410.246839:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017410.246845:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017410/real 1734017410] req@ffff8800a73cc380 x1818247332793984/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017410.246855:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332793984/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017410.246863:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017410.246865:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017410.246868:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332793984:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017410.246873:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332794112:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017410.246885:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017410.246891:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017410/real 1734017410] req@ffff8800a73cc700 x1818247332794112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017410.246902:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332794112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017410.246909:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017410.246911:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017410.246914:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332794112:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017410.246920:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332794240:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017410.246931:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017410.246937:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017410/real 1734017410] req@ffff8800a73ce680 x1818247332794240/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017410.246946:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332794240/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017410.246953:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017410.246955:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017410.246958:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332794240:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017410.246964:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332794368:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017410.246975:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017410.246981:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017410/real 1734017410] req@ffff8800a73cdc00 x1818247332794368/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017410.246991:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332794368/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017410.247006:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017410.247008:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017410.247011:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332794368:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017410.526509:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017411.528434:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017412.530499:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017413.532458:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017414.534509:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017415.254353:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017415.254362:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017415.254367:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017415.254377:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1677 00000100:00080000:3.0:1734017415.254388:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017415.254395:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017415.254433:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332794496/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017415.254467:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017415.254472:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017415.254474:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017415.254479:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1677 00000100:00080000:3.0:1734017415.254482:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017415.254486:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017415.254496:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017415.254508:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332794624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017415.254519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017415.254522:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017415.254524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017415.254528:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1677 00000100:00080000:3.0:1734017415.254530:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017415.254534:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017415.254538:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017415.254547:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332794752/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017415.254555:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017415.254558:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017415.254560:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017415.254563:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1677 00000100:00080000:3.0:1734017415.254566:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017415.254569:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017415.254573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017415.254582:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332794880/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017415.254583:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332794496:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017415.254591:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017415.254594:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017415.254595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017415.254599:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1677 00000100:00080000:3.0:1734017415.254602:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017415.254605:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017415.254609:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017415.254617:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332795008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017415.254625:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017415.254628:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017415.254629:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017415.254633:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1677 00000100:00080000:3.0:1734017415.254636:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017415.254639:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017415.254644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017415.254650:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332795136/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017415.254653:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017415.254658:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017415.254661:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017415.254663:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017415.254666:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1677 00000100:00080000:3.0:1734017415.254670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017415.254674:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017415.254675:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017415/real 1734017415] req@ffff8800a73cf100 x1818247332794496/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017415.254678:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017415.254686:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332795264/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017415.254694:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332794496/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017415.254705:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017415.254708:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017415.254713:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332794496:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017415.254720:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332794624:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017415.254736:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017415.254743:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017415/real 1734017415] req@ffff8800a73cdc00 x1818247332794624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017415.254754:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332794624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017415.254762:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017415.254765:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017415.254769:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332794624:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017415.254775:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332794752:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017415.254788:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017415.254794:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017415/real 1734017415] req@ffff8800a73ce680 x1818247332794752/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017415.254804:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332794752/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017415.254812:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017415.254815:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017415.254818:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332794752:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017415.254843:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332794880:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017415.254855:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017415.254861:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017415/real 1734017415] req@ffff8800a73cc700 x1818247332794880/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017415.254871:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332794880/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017415.254879:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017415.254881:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017415.254884:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332794880:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017415.254889:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332795008:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017415.254904:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017415.254910:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017415/real 1734017415] req@ffff8800a73cc380 x1818247332795008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017415.254921:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332795008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017415.254928:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017415.254930:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017415.254933:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332795008:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017415.254938:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332795136:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017415.254999:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017415.255008:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017415/real 1734017415] req@ffff8800a73cdf80 x1818247332795136/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017415.255019:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332795136/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017415.255026:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017415.255028:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017415.255031:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332795136:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017415.255037:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332795264:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017415.255050:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017415.255056:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017415/real 1734017415] req@ffff8800a73cf800 x1818247332795264/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017415.255073:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332795264/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017415.255081:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017415.255083:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017415.255086:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332795264:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017415.536479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017416.538496:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017417.540475:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017418.542495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017419.544501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017420.262382:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017420.262391:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017420.262395:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017420.262405:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1682 00000100:00080000:3.0:1734017420.262414:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017420.262422:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017420.262459:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332795392/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017420.262494:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017420.262498:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017420.262500:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017420.262504:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1682 00000100:00080000:3.0:1734017420.262508:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017420.262511:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017420.262521:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017420.262531:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332795520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017420.262541:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017420.262544:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017420.262546:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017420.262549:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1682 00000100:00080000:3.0:1734017420.262553:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017420.262557:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017420.262561:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017420.262570:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332795648/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017420.262578:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017420.262581:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017420.262582:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017420.262585:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1682 00000100:00080000:3.0:1734017420.262588:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017420.262591:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017420.262596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017420.262603:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332795776/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017420.262607:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332795392:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017420.262612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017420.262616:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017420.262617:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017420.262620:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1682 00000100:00080000:3.0:1734017420.262623:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017420.262626:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017420.262631:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017420.262638:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332795904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017420.262653:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017420.262656:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017420.262657:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017420.262661:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1682 00000100:00080000:3.0:1734017420.262664:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017420.262667:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017420.262672:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017420.262673:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017420.262679:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332796032/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017420.262687:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017420.262690:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017420.262691:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017420.262694:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017420/real 1734017420] req@ffff8800a73cfb80 x1818247332795392/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017420.262695:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1682 00000100:00080000:3.0:1734017420.262697:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017420.262700:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017420.262705:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017420.262713:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332796160/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017420.262714:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332795392/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017420.262724:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017420.262728:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017420.262732:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332795392:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017420.262739:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332795520:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017420.262757:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017420.262764:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017420/real 1734017420] req@ffff8800a73cca80 x1818247332795520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017420.262775:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332795520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017420.262783:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017420.262786:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017420.262789:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332795520:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017420.262795:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332795648:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017420.262808:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017420.262814:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017420/real 1734017420] req@ffff8800a73cf480 x1818247332795648/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017420.262825:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332795648/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017420.262832:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017420.262835:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017420.262838:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332795648:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017420.262912:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332795776:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017420.262930:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017420.262938:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017420/real 1734017420] req@ffff8800a73cd880 x1818247332795776/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017420.262948:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332795776/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017420.262956:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017420.262958:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017420.262961:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332795776:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017420.262968:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332795904:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017420.262980:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017420.262986:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017420/real 1734017420] req@ffff8800a73cea00 x1818247332795904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017420.262997:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332795904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017420.263005:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017420.263007:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017420.263009:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332795904:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017420.263014:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332796032:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017420.263026:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017420.263032:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017420/real 1734017420] req@ffff8800a73ced80 x1818247332796032/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017420.263042:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332796032/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017420.263049:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017420.263051:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017420.263053:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332796032:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017420.263059:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332796160:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017420.263071:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017420.263077:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017420/real 1734017420] req@ffff8800a73cce00 x1818247332796160/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017420.263088:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332796160/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017420.263095:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017420.263097:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017420.263100:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332796160:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017420.546480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017421.548468:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017422.550482:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017423.552467:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017424.554498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017425.270387:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017425.270396:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017425.270401:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017425.270412:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1687 00000100:00080000:3.0:1734017425.270422:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017425.270430:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017425.270467:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332796288/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017425.270502:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017425.270507:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017425.270509:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017425.270513:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1687 00000100:00080000:3.0:1734017425.270517:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017425.270520:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017425.270531:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017425.270543:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332796416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017425.270553:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017425.270556:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017425.270558:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017425.270562:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1687 00000100:00080000:3.0:1734017425.270564:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017425.270568:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017425.270573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017425.270581:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332796544/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017425.270590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017425.270593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017425.270595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017425.270598:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1687 00000100:00080000:3.0:1734017425.270601:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017425.270604:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017425.270608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017425.270616:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332796672/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017425.270622:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332796288:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017425.270624:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017425.270628:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017425.270629:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017425.270633:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1687 00000100:00080000:3.0:1734017425.270635:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017425.270638:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017425.270642:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017425.270650:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332796800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017425.270658:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017425.270661:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017425.270662:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017425.270666:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1687 00000100:00080000:3.0:1734017425.270669:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017425.270673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017425.270678:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017425.270685:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332796928/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017425.270690:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017425.270694:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017425.270697:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017425.270698:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017425.270701:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1687 00000100:00100000:1.0:1734017425.270712:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017425/real 1734017425] req@ffff8800a73cd500 x1818247332796288/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017425.270714:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017425.270717:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017425.270722:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017425.270730:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332797056/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017425.270731:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332796288/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017425.270742:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017425.270745:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017425.270750:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332796288:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017425.270758:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332796416:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017425.270817:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017425.270827:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017425/real 1734017425] req@ffff8800a73cce00 x1818247332796416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017425.270838:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332796416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017425.270847:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017425.270850:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017425.270854:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332796416:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017425.270860:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332796544:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017425.270877:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017425.270883:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017425/real 1734017425] req@ffff8800a73ced80 x1818247332796544/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017425.270894:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332796544/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017425.270901:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017425.270904:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017425.270907:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332796544:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017425.270931:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332796672:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017425.270943:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017425.270949:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017425/real 1734017425] req@ffff8800a73cea00 x1818247332796672/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017425.270960:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332796672/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017425.270967:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017425.270970:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017425.270973:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332796672:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017425.270979:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332796800:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017425.270991:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017425.270997:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017425/real 1734017425] req@ffff8800a73cd880 x1818247332796800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017425.271008:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332796800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017425.271016:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017425.271018:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017425.271021:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332796800:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017425.271026:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332796928:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017425.271038:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017425.271043:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017425/real 1734017425] req@ffff8800a73cf480 x1818247332796928/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017425.271053:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332796928/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017425.271060:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017425.271062:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017425.271065:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332796928:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017425.271070:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332797056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017425.271082:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017425.271088:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017425/real 1734017425] req@ffff8800a73cca80 x1818247332797056/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017425.271099:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332797056/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017425.271107:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017425.271109:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017425.271111:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332797056:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017425.556425:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017426.558485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017427.560447:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017428.562446:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017429.564443:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017430.278331:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017430.278340:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017430.278344:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017430.278354:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1692 00000100:00080000:3.0:1734017430.278364:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017430.278371:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017430.278407:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332797184/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017430.278443:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017430.278450:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017430.278452:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017430.278456:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1692 00000100:00080000:3.0:1734017430.278460:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017430.278463:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017430.278473:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017430.278484:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332797312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017430.278494:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017430.278497:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017430.278498:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017430.278502:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1692 00000100:00080000:3.0:1734017430.278505:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017430.278508:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017430.278513:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017430.278521:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332797440/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017430.278529:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017430.278532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017430.278534:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017430.278537:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1692 00000100:00080000:3.0:1734017430.278540:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017430.278543:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017430.278548:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017430.278554:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332797184:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017430.278556:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332797568/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017430.278565:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017430.278568:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017430.278569:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017430.278573:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1692 00000100:00080000:3.0:1734017430.278575:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017430.278578:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017430.278582:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017430.278590:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332797696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017430.278598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017430.278601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017430.278603:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017430.278606:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1692 00000100:00080000:3.0:1734017430.278610:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017430.278614:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017430.278618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017430.278620:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017430.278626:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332797824/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017430.278634:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017430.278637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017430.278639:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017430.278642:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017430/real 1734017430] req@ffff8800a73cfb80 x1818247332797184/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017430.278643:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1692 00000100:00080000:3.0:1734017430.278646:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017430.278649:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017430.278654:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017430.278662:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332797184/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017430.278662:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332797952/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017430.278673:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017430.278676:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017430.278681:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332797184:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017430.278688:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332797312:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017430.278748:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017430.278757:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017430/real 1734017430] req@ffff8800a73cf800 x1818247332797312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017430.278768:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332797312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017430.278777:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017430.278780:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017430.278784:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332797312:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017430.278790:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332797440:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017430.278804:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017430.278810:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017430/real 1734017430] req@ffff8800a73cdf80 x1818247332797440/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017430.278820:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332797440/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017430.278828:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017430.278830:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017430.278833:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332797440:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017430.278857:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332797568:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017430.278869:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017430.278875:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017430/real 1734017430] req@ffff8800a73cc380 x1818247332797568/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017430.278885:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332797568/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017430.278893:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017430.278895:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017430.278898:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332797568:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017430.278904:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332797696:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017430.278916:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017430.278922:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017430/real 1734017430] req@ffff8800a73cc700 x1818247332797696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017430.278932:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332797696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017430.278939:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017430.278941:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017430.278944:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332797696:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017430.278949:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332797824:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017430.278960:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017430.278966:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017430/real 1734017430] req@ffff8800a73ce680 x1818247332797824/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017430.278976:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332797824/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017430.278983:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017430.278986:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017430.278988:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332797824:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017430.278993:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332797952:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017430.279005:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017430.279011:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017430/real 1734017430] req@ffff8800a73cdc00 x1818247332797952/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017430.279022:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332797952/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017430.279029:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017430.279031:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017430.279034:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332797952:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017430.566483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017431.568445:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017432.570448:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017433.572404:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017434.574461:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017435.286346:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017435.286355:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017435.286359:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017435.286368:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1697 00000100:00080000:3.0:1734017435.286379:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017435.286386:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017435.286421:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332798080/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017435.286456:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017435.286460:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017435.286462:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017435.286467:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1697 00000100:00080000:3.0:1734017435.286470:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017435.286474:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017435.286483:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017435.286495:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332798208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017435.286506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017435.286509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017435.286511:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017435.286515:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1697 00000100:00080000:3.0:1734017435.286517:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017435.286520:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017435.286525:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017435.286533:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332798336/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017435.286541:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017435.286544:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017435.286546:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017435.286549:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1697 00000100:00080000:3.0:1734017435.286551:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017435.286555:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017435.286559:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017435.286564:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332798080:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017435.286567:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332798464/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017435.286576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017435.286579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017435.286581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017435.286584:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1697 00000100:00080000:3.0:1734017435.286587:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017435.286591:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017435.286595:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017435.286602:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332798592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017435.286644:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017435.286647:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017435.286649:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017435.286654:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1697 00000100:00080000:3.0:1734017435.286665:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017435.286669:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734017435.286672:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017435.286674:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017435.286684:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332798720/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017435.286692:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017435.286695:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017435/real 1734017435] req@ffff8800a73cf100 x1818247332798080/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017435.286695:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017435.286697:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017435.286701:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1697 00000100:00080000:3.0:1734017435.286703:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017435.286707:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017435.286711:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017435.286715:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332798080/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017435.286719:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332798848/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017435.286726:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017435.286729:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017435.286734:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332798080:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017435.286742:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332798208:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017435.286760:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017435.286767:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017435/real 1734017435] req@ffff8800a73cdc00 x1818247332798208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017435.286778:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332798208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017435.286787:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017435.286790:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017435.286793:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332798208:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017435.286799:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332798336:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017435.286812:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017435.286818:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017435/real 1734017435] req@ffff8800a73ce680 x1818247332798336/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017435.286828:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332798336/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017435.286835:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017435.286838:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017435.286841:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332798336:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017435.286865:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332798464:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017435.286877:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017435.286883:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017435/real 1734017435] req@ffff8800a73cc700 x1818247332798464/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017435.286894:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332798464/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017435.286901:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017435.286903:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017435.286906:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332798464:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017435.286913:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332798592:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017435.286925:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017435.286931:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017435/real 1734017435] req@ffff8800a73cc380 x1818247332798592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017435.286942:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332798592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017435.286949:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017435.286951:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017435.286954:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332798592:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017435.286960:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332798720:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017435.286971:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017435.286977:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017435/real 1734017435] req@ffff8800a73cdf80 x1818247332798720/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017435.286986:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332798720/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017435.286993:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017435.286996:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017435.286998:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332798720:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017435.287004:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332798848:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017435.287016:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017435.287022:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017435/real 1734017435] req@ffff8800a73cf800 x1818247332798848/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017435.287033:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332798848/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017435.287041:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017435.287043:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017435.287046:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332798848:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017435.576434:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017436.578481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017437.580439:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017438.582420:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017439.584443:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017440.294335:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017440.294343:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017440.294348:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017440.294358:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1702 00000100:00080000:3.0:1734017440.294369:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017440.294376:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017440.294412:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332798976/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017440.294448:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017440.294452:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017440.294454:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017440.294459:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1702 00000100:00080000:3.0:1734017440.294462:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017440.294466:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017440.294476:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017440.294486:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332799104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017440.294496:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017440.294499:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017440.294501:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017440.294505:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1702 00000100:00080000:3.0:1734017440.294508:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017440.294512:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017440.294516:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017440.294570:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332799232/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017440.294581:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017440.294584:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017440.294586:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017440.294589:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1702 00000100:00100000:1.0:1734017440.294592:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332798976:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017440.294593:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017440.294596:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017440.294601:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017440.294608:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332799360/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017440.294617:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017440.294620:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017440.294622:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017440.294625:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1702 00000100:00080000:3.0:1734017440.294627:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017440.294631:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017440.294635:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017440.294643:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332799488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017440.294658:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017440.294658:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017440.294661:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017440.294663:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017440.294666:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1702 00000100:00080000:3.0:1734017440.294669:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017440.294672:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017440.294677:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017440.294679:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017440/real 1734017440] req@ffff8800a73cfb80 x1818247332798976/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017440.294684:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332799616/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017440.294692:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017440.294695:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017440.294696:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017440.294697:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332798976/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017440.294700:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1702 00000100:00080000:3.0:1734017440.294702:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017440.294706:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:1.0:1734017440.294708:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1734017440.294710:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734017440.294711:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017440.294716:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332798976:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017440.294718:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332799744/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017440.294724:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332799104:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017440.294741:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017440.294747:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017440/real 1734017440] req@ffff8800a73cca80 x1818247332799104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017440.294758:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332799104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017440.294767:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017440.294770:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017440.294773:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332799104:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017440.294795:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332799232:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017440.294809:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017440.294815:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017440/real 1734017440] req@ffff8800a73cf480 x1818247332799232/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017440.294825:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332799232/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017440.294833:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017440.294835:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017440.294838:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332799232:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017440.294844:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332799360:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017440.294855:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017440.294861:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017440/real 1734017440] req@ffff8800a73cd880 x1818247332799360/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017440.294871:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332799360/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017440.294878:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017440.294880:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017440.294883:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332799360:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017440.294890:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332799488:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017440.294903:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017440.294909:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017440/real 1734017440] req@ffff8800a73cea00 x1818247332799488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017440.294920:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332799488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017440.294928:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017440.294930:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017440.294933:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332799488:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017440.294938:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332799616:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017440.294950:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017440.294956:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017440/real 1734017440] req@ffff8800a73ced80 x1818247332799616/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017440.294965:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332799616/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017440.294972:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017440.294975:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017440.294977:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332799616:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017440.294983:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332799744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017440.294996:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017440.295002:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017440/real 1734017440] req@ffff8800a73cce00 x1818247332799744/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017440.295013:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332799744/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017440.295022:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017440.295024:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017440.295027:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332799744:192.168.201.152@tcp:8:kworker.0 00010000:00010000:1.0:1734017440.586406:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017441.588422:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017442.590406:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017443.592493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1734017444.594494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017445.302339:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017445.302349:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017445.302353:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017445.302365:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1707 00000100:00080000:3.0:1734017445.302374:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017445.302383:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017445.302419:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332799872/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017445.302485:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017445.302489:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017445.302491:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017445.302496:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1707 00000100:00080000:3.0:1734017445.302500:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017445.302504:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017445.302514:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017445.302528:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332800000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017445.302538:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017445.302541:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017445.302543:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017445.302546:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1707 00000100:00080000:3.0:1734017445.302549:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017445.302553:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017445.302557:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017445.302565:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332800128/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017445.302574:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017445.302577:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017445.302579:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017445.302583:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1707 00000100:00080000:3.0:1734017445.302586:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017445.302589:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017445.302593:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017445.302599:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332799872:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017445.302601:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332800256/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017445.302610:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017445.302613:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017445.302615:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017445.302618:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1707 00000100:00080000:3.0:1734017445.302621:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017445.302624:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017445.302628:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017445.302635:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332800384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017445.302643:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017445.302646:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017445.302648:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017445.302652:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1707 00000100:00080000:3.0:1734017445.302654:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017445.302658:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017445.302663:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017445.302664:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017445.302671:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332800512/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017445.302680:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017445.302683:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017445.302684:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017445.302686:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017445/real 1734017445] req@ffff8800a73cd500 x1818247332799872/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017445.302688:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1707 00000100:00080000:3.0:1734017445.302691:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017445.302694:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017445.302698:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017445.302705:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332799872/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017445.302706:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332800640/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017445.302717:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017445.302720:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017445.302725:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332799872:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017445.302732:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332800000:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017445.302750:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017445.302756:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017445/real 1734017445] req@ffff8800a73cce00 x1818247332800000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017445.302768:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332800000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017445.302777:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017445.302780:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017445.302783:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332800000:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017445.302789:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332800128:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017445.302803:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017445.302809:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017445/real 1734017445] req@ffff8800a73ced80 x1818247332800128/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017445.302819:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332800128/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017445.302826:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017445.302829:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017445.302832:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332800128:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017445.302857:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332800256:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017445.302869:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017445.302876:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017445/real 1734017445] req@ffff8800a73cea00 x1818247332800256/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017445.302886:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332800256/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017445.302894:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017445.302896:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017445.302899:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332800256:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017445.302905:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332800384:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017445.302917:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017445.302923:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017445/real 1734017445] req@ffff8800a73cd880 x1818247332800384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017445.302934:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332800384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017445.302942:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017445.302944:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017445.302947:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332800384:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017445.302952:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332800512:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017445.302963:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017445.302969:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017445/real 1734017445] req@ffff8800a73cf480 x1818247332800512/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017445.302979:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332800512/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017445.302986:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017445.302988:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017445.302991:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332800512:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017445.302995:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332800640:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017445.303007:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017445.303012:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017445/real 1734017445] req@ffff8800a73cca80 x1818247332800640/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017445.303023:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332800640/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017445.303031:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017445.303033:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017445.303036:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332800640:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017445.596487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017446.598473:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017447.600465:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017448.602449:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017449.604455:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017450.310381:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017450.310392:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017450.310397:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017450.310407:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1712 00000100:00080000:3.0:1734017450.310417:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017450.310424:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017450.310462:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332800768/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017450.310497:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017450.310504:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017450.310506:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017450.310511:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1712 00000100:00080000:3.0:1734017450.310514:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017450.310518:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017450.310528:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017450.310538:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332800896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017450.310549:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017450.310552:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017450.310554:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017450.310557:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1712 00000100:00080000:3.0:1734017450.310560:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017450.310564:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017450.310569:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017450.310577:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332801024/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017450.310585:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017450.310588:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017450.310590:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017450.310593:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1712 00000100:00080000:3.0:1734017450.310595:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017450.310599:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017450.310604:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017450.310608:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332800768:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017450.310612:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332801152/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017450.310621:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017450.310624:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017450.310625:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017450.310629:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1712 00000100:00080000:3.0:1734017450.310631:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017450.310634:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017450.310639:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017450.310646:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332801280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017450.310654:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017450.310657:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017450.310659:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017450.310662:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1712 00000100:00080000:3.0:1734017450.310665:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017450.310668:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017450.310672:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017450.310675:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017450.310680:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332801408/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017450.310689:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017450.310692:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017450.310694:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017450.310697:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017450/real 1734017450] req@ffff8800a73cfb80 x1818247332800768/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017450.310698:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1712 00000100:00080000:3.0:1734017450.310701:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017450.310704:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017450.310708:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017450.310717:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332800768/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017450.310717:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332801536/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017450.310727:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017450.310730:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017450.310735:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332800768:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017450.310742:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332800896:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017450.310760:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017450.310766:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017450/real 1734017450] req@ffff8800a73cf800 x1818247332800896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017450.310778:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332800896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017450.310785:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017450.310789:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017450.310792:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332800896:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017450.310798:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332801024:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017450.310812:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017450.310817:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017450/real 1734017450] req@ffff8800a73cdf80 x1818247332801024/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017450.310827:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332801024/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017450.310835:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017450.310838:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017450.310841:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332801024:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017450.310865:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332801152:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017450.310877:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017450.310883:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017450/real 1734017450] req@ffff8800a73cc380 x1818247332801152/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017450.310893:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332801152/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017450.310901:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017450.310903:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017450.310906:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332801152:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017450.310911:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332801280:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017450.310926:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017450.310932:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017450/real 1734017450] req@ffff8800a73cc700 x1818247332801280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017450.310941:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332801280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017450.310948:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017450.310950:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017450.310953:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332801280:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017450.310958:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332801408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017450.310970:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017450.310975:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017450/real 1734017450] req@ffff8800a73ce680 x1818247332801408/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017450.310985:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332801408/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017450.310992:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017450.310994:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017450.310998:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332801408:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017450.311003:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332801536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017450.311014:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017450.311020:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017450/real 1734017450] req@ffff8800a73cdc00 x1818247332801536/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017450.311036:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332801536/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017450.311044:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017450.311046:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017450.311049:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332801536:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017450.606489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017451.608505:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017452.610507:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017453.612481:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017454.614485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017455.318394:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017455.318403:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017455.318407:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017455.318418:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1717 00000100:00080000:3.0:1734017455.318428:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017455.318436:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017455.318472:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332801664/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017455.318507:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017455.318512:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017455.318514:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017455.318518:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1717 00000100:00080000:3.0:1734017455.318521:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017455.318525:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017455.318535:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017455.318547:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332801792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017455.318558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017455.318561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017455.318563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017455.318566:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1717 00000100:00080000:3.0:1734017455.318569:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017455.318572:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017455.318577:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017455.318586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332801920/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017455.318594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017455.318597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017455.318598:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017455.318601:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1717 00000100:00080000:3.0:1734017455.318604:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017455.318607:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017455.318611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017455.318619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332802048/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017455.318625:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332801664:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017455.318627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017455.318631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017455.318632:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017455.318636:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1717 00000100:00080000:3.0:1734017455.318639:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017455.318642:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017455.318647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017455.318655:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332802176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017455.318663:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017455.318666:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017455.318667:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017455.318671:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1717 00000100:00080000:3.0:1734017455.318674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017455.318677:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017455.318681:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017455.318689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332802304/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017455.318691:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017455.318696:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017455.318699:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017455.318701:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017455.318704:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1717 00000100:00080000:3.0:1734017455.318707:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017455.318711:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017455.318713:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017455/real 1734017455] req@ffff8800a73cf100 x1818247332801664/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017455.318716:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017455.318724:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332802432/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017455.318732:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332801664/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017455.318742:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017455.318746:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017455.318750:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332801664:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017455.318757:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332801792:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017455.318774:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017455.318780:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017455/real 1734017455] req@ffff8800a73cdc00 x1818247332801792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017455.318791:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332801792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017455.318799:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017455.318802:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017455.318806:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332801792:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017455.318812:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332801920:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017455.318825:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017455.318831:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017455/real 1734017455] req@ffff8800a73ce680 x1818247332801920/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017455.318841:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332801920/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017455.318849:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017455.318851:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017455.318855:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332801920:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017455.318878:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332802048:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017455.318890:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017455.318896:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017455/real 1734017455] req@ffff8800a73cc700 x1818247332802048/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017455.318907:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332802048/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017455.318915:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017455.318917:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017455.318920:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332802048:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017455.318925:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332802176:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017455.318940:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017455.318946:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017455/real 1734017455] req@ffff8800a73cc380 x1818247332802176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017455.318956:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332802176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017455.318963:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017455.318966:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017455.318969:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332802176:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017455.318974:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332802304:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017455.318985:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017455.318991:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017455/real 1734017455] req@ffff8800a73cdf80 x1818247332802304/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017455.319000:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332802304/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017455.319008:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017455.319010:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017455.319013:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332802304:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017455.319018:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332802432:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017455.319030:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017455.319036:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017455/real 1734017455] req@ffff8800a73cf800 x1818247332802432/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017455.319052:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332802432/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017455.319059:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017455.319062:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017455.319065:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332802432:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017455.616470:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017456.618466:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017457.620478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017458.622429:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017459.624494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017460.326366:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017460.326375:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017460.326379:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017460.326390:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1722 00000100:00080000:3.0:1734017460.326400:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017460.326408:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017460.326443:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332802560/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017460.326477:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017460.326481:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017460.326483:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017460.326488:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1722 00000100:00080000:3.0:1734017460.326491:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017460.326495:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017460.326504:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017460.326515:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332802688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017460.326525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017460.326528:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017460.326530:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017460.326533:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1722 00000100:00080000:3.0:1734017460.326537:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017460.326541:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017460.326546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017460.326554:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332802816/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017460.326562:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017460.326565:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017460.326567:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017460.326570:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1722 00000100:00080000:3.0:1734017460.326572:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017460.326575:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017460.326580:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017460.326587:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332802944/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017460.326596:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332802560:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017460.326596:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017460.326599:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017460.326601:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017460.326604:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1722 00000100:00080000:3.0:1734017460.326606:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017460.326610:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017460.326614:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017460.326621:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332803072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017460.326637:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017460.326640:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017460.326641:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017460.326645:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1722 00000100:00080000:3.0:1734017460.326657:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017460.326661:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017460.326661:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017460.326665:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017460.326673:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332803200/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017460.326682:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017460.326683:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017460/real 1734017460] req@ffff8800a73cfb80 x1818247332802560/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017460.326685:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017460.326687:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017460.326690:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1722 00000100:00080000:3.0:1734017460.326693:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017460.326696:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017460.326700:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017460.326701:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332802560/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017460.326709:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332803328/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017460.326712:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017460.326715:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017460.326720:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332802560:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017460.326727:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332802688:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017460.326744:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017460.326750:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017460/real 1734017460] req@ffff8800a73cca80 x1818247332802688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017460.326761:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332802688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017460.326769:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017460.326773:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017460.326776:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332802688:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017460.326782:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332802816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017460.326796:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017460.326802:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017460/real 1734017460] req@ffff8800a73cf480 x1818247332802816/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017460.326812:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332802816/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017460.326819:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017460.326822:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017460.326825:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332802816:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017460.326849:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332802944:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017460.326861:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017460.326867:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017460/real 1734017460] req@ffff8800a73cd880 x1818247332802944/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017460.326878:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332802944/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017460.326886:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017460.326888:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017460.326891:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332802944:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017460.326896:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332803072:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017460.326911:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017460.326917:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017460/real 1734017460] req@ffff8800a73cea00 x1818247332803072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017460.326928:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332803072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017460.326935:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017460.326937:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017460.326940:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332803072:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017460.326946:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332803200:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017460.326957:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017460.326963:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017460/real 1734017460] req@ffff8800a73ced80 x1818247332803200/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017460.326973:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332803200/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017460.326981:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017460.326983:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017460.326986:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332803200:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017460.326991:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332803328:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017460.327003:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017460.327008:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017460/real 1734017460] req@ffff8800a73cce00 x1818247332803328/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017460.327025:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332803328/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017460.327033:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017460.327035:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017460.327038:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332803328:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017460.626419:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017461.628494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017462.630467:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017463.632487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017464.634448:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017465.334360:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017465.334369:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017465.334374:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017465.334386:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1727 00000100:00080000:3.0:1734017465.334395:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017465.334404:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017465.334439:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332803456/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017465.334474:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017465.334478:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017465.334481:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017465.334485:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1727 00000100:00080000:3.0:1734017465.334488:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017465.334492:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017465.334502:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017465.334514:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332803584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017465.334524:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017465.334527:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017465.334529:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017465.334533:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1727 00000100:00080000:3.0:1734017465.334535:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017465.334539:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017465.334543:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017465.334552:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332803712/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017465.334560:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017465.334564:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017465.334566:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017465.334569:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1727 00000100:00080000:3.0:1734017465.334572:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017465.334575:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017465.334579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017465.334587:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332803456:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017465.334587:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332803840/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017465.334596:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017465.334599:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017465.334601:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017465.334604:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1727 00000100:00080000:3.0:1734017465.334606:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017465.334609:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017465.334614:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017465.334621:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332803968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017465.334629:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017465.334633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017465.334634:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017465.334638:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1727 00000100:00080000:3.0:1734017465.334649:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017465.334653:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017465.334653:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017465.334658:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017465.334666:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332804096/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017465.334675:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017465/real 1734017465] req@ffff8800a73cd500 x1818247332803456/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017465.334675:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017465.334678:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017465.334680:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017465.334683:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1727 00000100:00080000:3.0:1734017465.334686:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017465.334689:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017465.334693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017465.334694:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332803456/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017465.334701:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332804224/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017465.334705:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017465.334708:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017465.334713:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332803456:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017465.334721:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332803584:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017465.334738:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017465.334744:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017465/real 1734017465] req@ffff8800a73cce00 x1818247332803584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017465.334755:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332803584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017465.334763:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017465.334767:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017465.334770:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332803584:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017465.334776:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332803712:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017465.334790:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017465.334796:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017465/real 1734017465] req@ffff8800a73ced80 x1818247332803712/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017465.334806:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332803712/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017465.334813:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017465.334816:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017465.334819:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332803712:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017465.334844:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332803840:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017465.334857:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017465.334862:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017465/real 1734017465] req@ffff8800a73cea00 x1818247332803840/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017465.334873:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332803840/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017465.334880:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017465.334882:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017465.334885:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332803840:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017465.334891:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332803968:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017465.334906:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017465.334912:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017465/real 1734017465] req@ffff8800a73cd880 x1818247332803968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017465.334922:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332803968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017465.334929:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017465.334931:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017465.334934:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332803968:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017465.334940:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332804096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017465.334951:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017465.334957:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017465/real 1734017465] req@ffff8800a73cf480 x1818247332804096/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017465.334967:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332804096/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017465.334974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017465.334976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017465.334980:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332804096:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017465.334985:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332804224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017465.334997:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017465.335002:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017465/real 1734017465] req@ffff8800a73cca80 x1818247332804224/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017465.335019:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332804224/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017465.335026:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017465.335028:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017465.335031:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332804224:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017465.636469:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017466.638483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017467.640477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017468.642455:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017469.644489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017470.342351:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017470.342360:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017470.342365:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017470.342376:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1732 00000100:00080000:3.0:1734017470.342385:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017470.342392:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017470.342428:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332804352/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017470.342463:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017470.342470:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017470.342472:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017470.342476:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1732 00000100:00080000:3.0:1734017470.342479:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017470.342483:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017470.342493:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017470.342504:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332804480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017470.342514:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017470.342517:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017470.342519:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017470.342523:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1732 00000100:00080000:3.0:1734017470.342525:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017470.342529:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017470.342534:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017470.342542:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332804608/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017470.342550:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017470.342553:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017470.342555:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017470.342558:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1732 00000100:00080000:3.0:1734017470.342561:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017470.342564:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017470.342569:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017470.342574:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332804352:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017470.342577:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332804736/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017470.342587:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017470.342590:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017470.342592:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017470.342595:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1732 00000100:00080000:3.0:1734017470.342597:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017470.342600:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017470.342605:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017470.342612:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332804864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017470.342620:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017470.342623:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017470.342625:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017470.342629:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1732 00000100:00080000:3.0:1734017470.342631:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017470.342635:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017470.342639:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017470.342646:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332804992/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017470.342655:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017470.342658:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017470.342660:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734017470.342663:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332804480:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017470.342664:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1732 00000100:00080000:3.0:1734017470.342666:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017470.342670:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017470.342674:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017470.342682:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332805120/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017470.342684:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332804608:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017470.342707:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332804736:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017470.342721:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332804864:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017470.342736:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332804992:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017470.342750:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332805120:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017470.646475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017471.648497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017472.650475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734017473.348517:0:1955:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734017473.348578:0:1955:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734017473.348603:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017473.348650:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017473.348660:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017473.348669:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017473.348676:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017473.348683:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017473.348690:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734017473.348765:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017470/real 1734017473] req@ffff8800a73cc380 x1818247332804736/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017473.348798:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332804736/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017473.348811:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017473.348815:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017473.348820:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332804736:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017473.348829:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017470/real 1734017473] req@ffff8800a73cc700 x1818247332804864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017473.348849:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332804864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017473.348862:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017473.348866:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017473.348871:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332804864:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017473.348882:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017470/real 1734017473] req@ffff8800a73ce680 x1818247332804992/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017473.348901:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332804992/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017473.348913:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017473.348917:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017473.348922:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332804992:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017473.348933:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017470/real 1734017473] req@ffff8800a73cdc00 x1818247332805120/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017473.348951:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332805120/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017473.349029:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017473.349034:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017473.349040:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332805120:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017473.349056:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017470/real 1734017473] req@ffff8800a73cfb80 x1818247332804352/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017473.349077:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332804352/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017473.349089:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017473.349093:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017473.349098:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332804352:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017473.349109:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017470/real 1734017473] req@ffff8800a73cf800 x1818247332804480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017473.349127:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332804480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017473.349140:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017473.349144:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017473.349149:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332804480:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017473.349159:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017470/real 1734017473] req@ffff8800a73cdf80 x1818247332804608/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017473.349177:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332804608/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017473.349190:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017473.349194:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017473.349199:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332804608:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017473.652492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017474.654484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017475.350382:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017475.350392:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017475.350397:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017475.350407:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1737 00000100:00080000:3.0:1734017475.350418:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017475.350426:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017475.350462:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332805248/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017475.350498:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017475.350503:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017475.350505:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017475.350509:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1737 00000100:00080000:3.0:1734017475.350512:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017475.350516:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017475.350527:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017475.350539:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332805376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017475.350550:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017475.350554:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017475.350555:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017475.350559:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1737 00000100:00080000:3.0:1734017475.350561:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017475.350565:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017475.350569:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017475.350578:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332805504/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017475.350586:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017475.350589:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017475.350590:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017475.350594:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1737 00000100:00080000:3.0:1734017475.350596:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017475.350600:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017475.350604:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017475.350612:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332805632/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017475.350620:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332805248:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017475.350621:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017475.350624:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017475.350626:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017475.350629:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1737 00000100:00080000:3.0:1734017475.350633:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017475.350636:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017475.350640:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017475.350648:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332805760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017475.350656:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017475.350659:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017475.350660:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017475.350664:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1737 00000100:00080000:3.0:1734017475.350666:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017475.350669:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017475.350673:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017475.350680:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332805888/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017475.350688:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734017475.350689:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017475.350691:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017475.350693:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017475.350696:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1737 00000100:00080000:3.0:1734017475.350698:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017475.350701:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017475.350706:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017475.350710:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017475/real 1734017475] req@ffff8800a73cf100 x1818247332805248/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017475.350713:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332806016/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017475.350730:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332805248/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017475.350741:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017475.350744:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017475.350749:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332805248:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017475.350756:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332805376:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017475.350773:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017475.350779:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017475/real 1734017475] req@ffff8800a73cdf80 x1818247332805376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017475.350791:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332805376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017475.350799:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017475.350801:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017475.350805:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332805376:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017475.350811:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332805504:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017475.350824:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017475.350830:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017475/real 1734017475] req@ffff8800a73cf800 x1818247332805504/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017475.350840:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332805504/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017475.350849:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017475.350851:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017475.350855:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332805504:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017475.350879:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332805632:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017475.350891:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017475.350897:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017475/real 1734017475] req@ffff8800a73cfb80 x1818247332805632/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017475.350907:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332805632/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017475.350915:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017475.350917:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017475.350920:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332805632:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017475.350972:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332805760:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017475.350989:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017475.350996:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017475/real 1734017475] req@ffff8800a73cdc00 x1818247332805760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017475.351006:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332805760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017475.351014:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017475.351016:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017475.351019:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332805760:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017475.351025:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332805888:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017475.351037:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017475.351043:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017475/real 1734017475] req@ffff8800a73ce680 x1818247332805888/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017475.351054:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332805888/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017475.351061:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017475.351063:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017475.351066:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332805888:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017475.351071:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332806016:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017475.351083:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017475.351088:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017475/real 1734017475] req@ffff8800a73cc700 x1818247332806016/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017475.351098:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332806016/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017475.351105:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017475.351107:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017475.351110:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332806016:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017475.656464:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017476.658469:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017477.660443:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017478.662483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017479.664461:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017480.358327:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017480.358333:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017480.358336:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017480.358343:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1742 00000100:00080000:3.0:1734017480.358351:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017480.358357:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017480.358381:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332806144/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017480.358405:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017480.358408:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017480.358409:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017480.358411:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1742 00000100:00080000:3.0:1734017480.358413:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017480.358415:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017480.358421:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017480.358427:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332806272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017480.358432:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017480.358434:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017480.358435:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017480.358437:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1742 00000100:00080000:3.0:1734017480.358439:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017480.358441:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017480.358444:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017480.358448:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332806400/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017480.358452:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017480.358454:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017480.358455:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017480.358456:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1742 00000100:00080000:3.0:1734017480.358458:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017480.358459:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017480.358462:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017480.358466:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332806528/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017480.358470:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017480.358471:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017480.358472:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017480.358474:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1742 00000100:00080000:3.0:1734017480.358475:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017480.358477:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017480.358479:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017480.358484:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332806656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017480.358486:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332806144:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017480.358492:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017480.358494:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017480.358495:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017480.358497:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1742 00000100:00080000:3.0:1734017480.358498:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017480.358500:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017480.358502:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017480.358506:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332806784/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017480.358510:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017480.358511:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017480.358512:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017480.358514:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1742 00000100:00080000:3.0:1734017480.358515:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017480.358517:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017480.358519:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017480.358524:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332806912/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017480.358531:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017480.358546:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017480/real 1734017480] req@ffff8800a73cc380 x1818247332806144/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017480.358557:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332806144/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017480.358563:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017480.358564:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017480.358567:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332806144:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017480.358571:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332806272:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017480.358579:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017480.358583:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017480/real 1734017480] req@ffff8800a73cca80 x1818247332806272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017480.358588:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332806272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017480.358592:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017480.358594:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017480.358595:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332806272:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017480.358598:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332806400:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017480.358605:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017480.358608:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017480/real 1734017480] req@ffff8800a73cf480 x1818247332806400/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017480.358613:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332806400/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017480.358617:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017480.358619:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017480.358621:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332806400:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017480.358623:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332806528:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017480.358630:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017480.358633:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017480/real 1734017480] req@ffff8800a73cd880 x1818247332806528/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017480.358638:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332806528/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017480.358641:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017480.358642:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017480.358644:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332806528:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017480.358659:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332806656:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017480.358666:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017480.358669:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017480/real 1734017480] req@ffff8800a73cea00 x1818247332806656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017480.358674:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332806656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017480.358677:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017480.358678:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017480.358680:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332806656:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017480.358683:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332806784:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017480.358691:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017480.358694:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017480/real 1734017480] req@ffff8800a73ced80 x1818247332806784/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017480.358700:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332806784/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017480.358703:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017480.358704:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017480.358706:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332806784:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017480.358708:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332806912:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017480.358714:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017480.358717:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017480/real 1734017480] req@ffff8800a73cce00 x1818247332806912/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017480.358722:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332806912/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017480.358725:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017480.358726:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017480.358728:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332806912:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017480.666457:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017481.668453:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017482.670481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017483.672549:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017484.674484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017485.366358:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017485.366366:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017485.366371:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017485.366382:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1747 00000100:00080000:3.0:1734017485.366392:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017485.366400:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017485.366437:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332807040/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017485.366470:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017485.366475:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017485.366477:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017485.366482:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1747 00000100:00080000:3.0:1734017485.366484:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017485.366489:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017485.366499:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017485.366511:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332807168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017485.366522:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017485.366525:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017485.366527:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017485.366530:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1747 00000100:00080000:3.0:1734017485.366533:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017485.366536:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017485.366541:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017485.366549:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332807296/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017485.366558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017485.366561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017485.366563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017485.366567:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1747 00000100:00080000:3.0:1734017485.366569:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017485.366572:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017485.366577:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017485.366583:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332807040:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017485.366584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332807424/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017485.366593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017485.366597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017485.366598:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017485.366601:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1747 00000100:00080000:3.0:1734017485.366604:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017485.366607:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017485.366611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017485.366618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332807552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017485.366626:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017485.366629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017485.366631:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017485.366635:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1747 00000100:00080000:3.0:1734017485.366646:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017485.366650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734017485.366651:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017485.366655:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017485.366663:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332807680/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017485.366672:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017485.366675:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017485.366676:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017485.366680:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1747 00000100:00080000:3.0:1734017485.366684:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017485.366688:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017485.366689:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017485/real 1734017485] req@ffff8800a73cd500 x1818247332807040/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017485.366692:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017485.366700:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332807808/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017485.366716:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332807040/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017485.366732:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017485.366735:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017485.366740:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332807040:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017485.366748:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332807168:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017485.366811:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017485.366819:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017485/real 1734017485] req@ffff8800a73cce00 x1818247332807168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017485.366831:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332807168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017485.366839:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017485.366842:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017485.366845:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332807168:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017485.366852:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332807296:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017485.366865:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017485.366871:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017485/real 1734017485] req@ffff8800a73ced80 x1818247332807296/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017485.366881:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332807296/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017485.366889:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017485.366892:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017485.366895:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332807296:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017485.366920:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332807424:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017485.366932:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017485.366939:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017485/real 1734017485] req@ffff8800a73cea00 x1818247332807424/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017485.366949:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332807424/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017485.366956:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017485.366958:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017485.366961:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332807424:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017485.366966:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332807552:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017485.366978:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017485.366984:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017485/real 1734017485] req@ffff8800a73cd880 x1818247332807552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017485.366993:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332807552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017485.367001:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017485.367004:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017485.367007:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332807552:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017485.367012:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332807680:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017485.367027:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017485.367033:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017485/real 1734017485] req@ffff8800a73cf480 x1818247332807680/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017485.367043:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332807680/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017485.367050:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017485.367052:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017485.367055:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332807680:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017485.367060:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332807808:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017485.367071:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017485.367077:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017485/real 1734017485] req@ffff8800a73cca80 x1818247332807808/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017485.367087:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332807808/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017485.367094:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017485.367096:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017485.367099:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332807808:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017485.676428:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017486.678503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017487.680465:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017488.682464:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017489.684459:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017490.374367:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017490.374376:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017490.374380:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017490.374391:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1752 00000100:00080000:3.0:1734017490.374401:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017490.374409:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017490.374445:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332807936/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017490.374480:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017490.374487:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017490.374489:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017490.374494:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1752 00000100:00080000:3.0:1734017490.374497:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017490.374501:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017490.374511:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017490.374521:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332808064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017490.374531:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017490.374534:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017490.374536:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017490.374539:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1752 00000100:00080000:3.0:1734017490.374542:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017490.374546:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017490.374550:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017490.374558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332808192/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017490.374567:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017490.374570:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017490.374571:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017490.374575:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1752 00000100:00080000:3.0:1734017490.374577:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017490.374580:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017490.374585:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017490.374593:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332808320/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017490.374599:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332807936:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017490.374602:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017490.374606:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017490.374607:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017490.374611:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1752 00000100:00080000:3.0:1734017490.374613:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017490.374616:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017490.374621:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017490.374628:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332808448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017490.374636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017490.374639:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017490.374641:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017490.374644:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1752 00000100:00080000:3.0:1734017490.374647:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017490.374650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017490.374654:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017490.374661:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332808576/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017490.374703:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017490.374703:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017490.374706:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017490.374709:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017490.374713:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1752 00000100:00080000:3.0:1734017490.374716:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017490.374719:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017490.374724:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017490.374728:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017490/real 1734017490] req@ffff8800a73cc380 x1818247332807936/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017490.374735:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332808704/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017490.374756:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332807936/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017490.374766:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017490.374770:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017490.374775:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332807936:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017490.374782:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332808064:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017490.374802:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017490.374808:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017490/real 1734017490] req@ffff8800a73cc700 x1818247332808064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017490.374819:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332808064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017490.374827:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017490.374830:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017490.374833:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332808064:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017490.374839:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332808192:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017490.374852:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017490.374858:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017490/real 1734017490] req@ffff8800a73ce680 x1818247332808192/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017490.374867:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332808192/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017490.374875:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017490.374878:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017490.374881:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332808192:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017490.374905:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332808320:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017490.374918:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017490.374924:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017490/real 1734017490] req@ffff8800a73cdc00 x1818247332808320/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017490.374935:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332808320/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017490.374942:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017490.374944:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017490.374947:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332808320:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017490.374952:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332808448:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017490.374964:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017490.374970:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017490/real 1734017490] req@ffff8800a73cfb80 x1818247332808448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017490.374980:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332808448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017490.374987:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017490.374989:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017490.374992:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332808448:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017490.374998:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332808576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017490.375013:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017490.375019:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017490/real 1734017490] req@ffff8800a73cf800 x1818247332808576/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017490.375029:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332808576/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017490.375036:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017490.375038:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017490.375041:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332808576:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017490.375046:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332808704:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017490.375058:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017490.375063:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017490/real 1734017490] req@ffff8800a73cdf80 x1818247332808704/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017490.375073:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332808704/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017490.375080:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017490.375082:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017490.375085:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332808704:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017490.686463:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017491.688486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017492.690510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017493.692452:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017494.694455:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017495.382343:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017495.382352:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017495.382356:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017495.382366:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1757 00000100:00080000:3.0:1734017495.382375:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017495.382383:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017495.382418:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332808832/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017495.382452:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017495.382456:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017495.382458:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017495.382463:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1757 00000100:00080000:3.0:1734017495.382467:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017495.382470:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017495.382481:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017495.382493:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332808960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017495.382504:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017495.382507:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017495.382509:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017495.382512:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1757 00000100:00080000:3.0:1734017495.382515:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017495.382518:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017495.382523:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017495.382530:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332809088/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017495.382539:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017495.382542:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017495.382543:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017495.382546:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1757 00000100:00080000:3.0:1734017495.382549:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017495.382552:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017495.382556:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017495.382564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332809216/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017495.382573:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017495.382576:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332808832:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017495.382576:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017495.382577:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017495.382613:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1757 00000100:00080000:3.0:1734017495.382617:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017495.382621:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017495.382626:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017495.382635:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332809344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017495.382644:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017495.382648:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017495.382649:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017495.382653:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1757 00000100:00080000:3.0:1734017495.382656:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017495.382659:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017495.382663:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017495.382671:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332809472/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017495.382678:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017495.382682:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017495.382683:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734017495.382685:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017495.382687:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1757 00000100:00080000:3.0:1734017495.382690:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017495.382693:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017495.382697:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017495.382705:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332809600/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017495.382707:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017495/real 1734017495] req@ffff8800a73cf100 x1818247332808832/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017495.382734:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332808832/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017495.382744:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017495.382748:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017495.382752:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332808832:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017495.382759:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332808960:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017495.382775:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017495.382782:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017495/real 1734017495] req@ffff8800a73cdf80 x1818247332808960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017495.382793:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332808960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017495.382800:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017495.382803:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017495.382806:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332808960:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017495.382812:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332809088:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017495.382825:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017495.382831:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017495/real 1734017495] req@ffff8800a73cf800 x1818247332809088/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017495.382840:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332809088/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017495.382848:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017495.382851:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017495.382854:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332809088:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017495.382879:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332809216:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017495.382891:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017495.382897:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017495/real 1734017495] req@ffff8800a73cfb80 x1818247332809216/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017495.382908:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332809216/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017495.382915:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017495.382917:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017495.382920:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332809216:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017495.382925:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332809344:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017495.382937:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017495.382942:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017495/real 1734017495] req@ffff8800a73cdc00 x1818247332809344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017495.382952:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332809344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017495.382959:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017495.382961:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017495.382965:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332809344:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017495.382971:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332809472:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017495.382985:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017495.382991:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017495/real 1734017495] req@ffff8800a73ce680 x1818247332809472/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017495.383002:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332809472/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017495.383009:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017495.383011:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017495.383013:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332809472:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017495.383019:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332809600:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017495.383030:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017495.383036:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017495/real 1734017495] req@ffff8800a73cc700 x1818247332809600/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017495.383046:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332809600/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017495.383053:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017495.383055:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017495.383058:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332809600:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017495.696437:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017496.698462:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017497.700486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017498.702446:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017499.704490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017500.390322:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017500.390331:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017500.390335:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017500.390346:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1762 00000100:00080000:3.0:1734017500.390355:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017500.390363:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017500.390399:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332809728/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017500.390434:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017500.390438:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017500.390440:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017500.390444:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1762 00000100:00080000:3.0:1734017500.390447:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017500.390451:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017500.390461:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017500.390471:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332809856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017500.390481:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017500.390485:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017500.390486:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017500.390490:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1762 00000100:00080000:3.0:1734017500.390493:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017500.390533:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017500.390539:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017500.390548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332809984/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017500.390557:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017500.390561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017500.390562:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017500.390565:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1762 00000100:00080000:3.0:1734017500.390568:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017500.390571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017500.390576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017500.390584:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332809728:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017500.390584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332810112/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017500.390593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017500.390596:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017500.390598:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017500.390602:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1762 00000100:00080000:3.0:1734017500.390604:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017500.390608:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017500.390612:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017500.390619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332810240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017500.390634:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017500.390637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017500.390639:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017500.390643:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1762 00000100:00080000:3.0:1734017500.390645:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017500.390649:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734017500.390650:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017500.390653:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017500.390660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332810368/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017500.390668:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017500.390671:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734017500.390672:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017500/real 1734017500] req@ffff8800a73cc380 x1818247332809728/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017500.390672:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017500.390676:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1762 00000100:00080000:3.0:1734017500.390678:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017500.390681:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017500.390685:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017500.390693:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332810496/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017500.390699:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332809728/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017500.390709:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017500.390712:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017500.390717:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332809728:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017500.390724:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332809856:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017500.390741:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017500.390748:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017500/real 1734017500] req@ffff8800a73cca80 x1818247332809856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017500.390759:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332809856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017500.390766:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017500.390769:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017500.390772:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332809856:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017500.390778:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332809984:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017500.390791:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017500.390797:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017500/real 1734017500] req@ffff8800a73cf480 x1818247332809984/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017500.390807:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332809984/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017500.390814:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017500.390817:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017500.390820:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332809984:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017500.390843:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332810112:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017500.390856:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017500.390862:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017500/real 1734017500] req@ffff8800a73cd880 x1818247332810112/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017500.390873:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332810112/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017500.390880:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017500.390882:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017500.390885:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332810112:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017500.390891:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332810240:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017500.390903:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017500.390909:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017500/real 1734017500] req@ffff8800a73cea00 x1818247332810240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017500.390918:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332810240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017500.390926:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017500.390928:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017500.390931:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332810240:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017500.390937:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332810368:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017500.390952:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017500.390958:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017500/real 1734017500] req@ffff8800a73ced80 x1818247332810368/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017500.390968:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332810368/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017500.390975:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017500.390977:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017500.390980:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332810368:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017500.390986:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332810496:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017500.390997:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017500.391003:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017500/real 1734017500] req@ffff8800a73cce00 x1818247332810496/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017500.391013:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332810496/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017500.391020:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017500.391022:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017500.391025:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332810496:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017500.706477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017501.708501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017502.710446:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017503.712386:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017504.714486:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017505.398365:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017505.398373:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017505.398378:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017505.398388:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1767 00000100:00080000:3.0:1734017505.398397:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017505.398405:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017505.398468:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332810624/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017505.398500:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017505.398504:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017505.398506:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017505.398511:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1767 00000100:00080000:3.0:1734017505.398514:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017505.398518:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017505.398528:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017505.398541:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332810752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017505.398552:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017505.398555:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017505.398557:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017505.398560:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1767 00000100:00080000:3.0:1734017505.398563:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017505.398566:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017505.398571:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017505.398579:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332810880/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017505.398588:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017505.398592:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017505.398594:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017505.398597:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1767 00000100:00080000:3.0:1734017505.398599:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017505.398603:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017505.398607:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017505.398615:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332811008/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017505.398616:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332810624:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017505.398624:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017505.398627:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017505.398628:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017505.398632:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1767 00000100:00080000:3.0:1734017505.398634:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017505.398637:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017505.398642:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017505.398649:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332811136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017505.398657:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017505.398660:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017505.398662:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017505.398665:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1767 00000100:00080000:3.0:1734017505.398668:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017505.398671:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017505.398676:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017505.398682:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017505.398684:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332811264/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017505.398693:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017505.398696:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017505.398697:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017505.398700:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1767 00000100:00080000:3.0:1734017505.398703:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017505.398705:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017505/real 1734017505] req@ffff8800a73cd500 x1818247332810624/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017505.398706:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017505.398710:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017505.398718:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332811392/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017505.398732:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332810624/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017505.398742:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017505.398745:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017505.398750:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332810624:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017505.398757:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332810752:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017505.398774:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017505.398781:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017505/real 1734017505] req@ffff8800a73cce00 x1818247332810752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017505.398792:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332810752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017505.398800:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017505.398802:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017505.398806:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332810752:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017505.398811:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332810880:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017505.398825:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017505.398831:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017505/real 1734017505] req@ffff8800a73ced80 x1818247332810880/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017505.398841:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332810880/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017505.398848:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017505.398851:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017505.398854:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332810880:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017505.398880:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332811008:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017505.398892:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017505.398899:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017505/real 1734017505] req@ffff8800a73cea00 x1818247332811008/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017505.398909:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332811008/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017505.398917:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017505.398919:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017505.398922:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332811008:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017505.398927:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332811136:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017505.398939:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017505.398945:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017505/real 1734017505] req@ffff8800a73cd880 x1818247332811136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017505.398954:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332811136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017505.398962:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017505.398964:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017505.398967:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332811136:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017505.398973:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332811264:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017505.398988:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017505.398994:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017505/real 1734017505] req@ffff8800a73cf480 x1818247332811264/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017505.399004:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332811264/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017505.399011:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017505.399013:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017505.399016:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332811264:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017505.399021:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332811392:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017505.399032:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017505.399038:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017505/real 1734017505] req@ffff8800a73cca80 x1818247332811392/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017505.399047:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332811392/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017505.399054:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017505.399056:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017505.399059:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332811392:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017505.716477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017506.718474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017507.720474:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017508.722439:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017509.724534:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017510.406387:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017510.406397:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017510.406401:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017510.406411:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1772 00000100:00080000:3.0:1734017510.406420:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017510.406428:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017510.406464:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332811520/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017510.406496:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017510.406504:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017510.406506:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017510.406510:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1772 00000100:00080000:3.0:1734017510.406513:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017510.406517:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017510.406526:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017510.406536:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332811648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017510.406546:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017510.406549:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017510.406551:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017510.406554:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1772 00000100:00080000:3.0:1734017510.406557:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017510.406560:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017510.406565:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017510.406573:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332811776/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017510.406579:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332811520:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017510.406582:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017510.406585:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017510.406587:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017510.406590:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1772 00000100:00080000:3.0:1734017510.406593:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017510.406596:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017510.406601:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017510.406609:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332811904/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017510.406617:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017510.406620:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017510.406622:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017510.406625:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1772 00000100:00080000:3.0:1734017510.406627:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017510.406631:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017510.406635:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017510.406642:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332812032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017510.406646:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017510.406650:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017510.406653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017510.406655:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017510.406658:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1772 00000100:00100000:1.0:1734017510.406668:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017510/real 1734017510] req@ffff8800a73cc380 x1818247332811520/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017510.406670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017510.406674:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017510.406678:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017510.406686:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332812160/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017510.406694:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017510.406695:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332811520/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017510.406697:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017510.406699:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017510.406703:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1772 00000100:00080000:1.0:1734017510.406705:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017510.406706:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1734017510.406708:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734017510.406709:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734017510.406713:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332811520:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017510.406713:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017510.406720:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332811648:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734017510.406722:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332812288/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017510.406738:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017510.406744:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017510/real 1734017510] req@ffff8800a73cc700 x1818247332811648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017510.406755:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332811648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017510.406763:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017510.406766:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017510.406769:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332811648:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017510.406790:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332811776:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017510.406804:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017510.406810:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017510/real 1734017510] req@ffff8800a73ce680 x1818247332811776/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017510.406820:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332811776/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017510.406827:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017510.406831:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017510.406834:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332811776:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017510.406840:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332811904:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017510.406852:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017510.406857:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017510/real 1734017510] req@ffff8800a73cdc00 x1818247332811904/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017510.406867:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332811904/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017510.406875:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017510.406877:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017510.406880:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332811904:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017510.406885:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332812032:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017510.406897:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017510.406903:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017510/real 1734017510] req@ffff8800a73cfb80 x1818247332812032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017510.406913:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332812032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017510.406920:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017510.406922:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017510.406926:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332812032:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017510.406931:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332812160:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017510.406946:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017510.406952:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017510/real 1734017510] req@ffff8800a73cf800 x1818247332812160/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017510.406962:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332812160/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017510.406969:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017510.406972:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017510.406975:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332812160:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017510.406980:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332812288:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017510.406992:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017510.406997:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017510/real 1734017510] req@ffff8800a73cdf80 x1818247332812288/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017510.407007:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332812288/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017510.407014:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017510.407016:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017510.407019:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332812288:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017510.726504:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017511.728524:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017512.730494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017513.732471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017514.734477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017515.414359:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017515.414368:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017515.414373:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017515.414383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1777 00000100:00080000:3.0:1734017515.414391:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017515.414399:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017515.414433:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332812416/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017515.414468:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017515.414472:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017515.414474:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017515.414478:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1777 00000100:00080000:3.0:1734017515.414481:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017515.414485:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017515.414495:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017515.414507:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332812544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017515.414519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017515.414522:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017515.414524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017515.414527:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1777 00000100:00080000:3.0:1734017515.414530:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017515.414533:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017515.414538:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017515.414545:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332812672/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017515.414554:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017515.414557:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017515.414558:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017515.414562:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1777 00000100:00080000:3.0:1734017515.414564:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017515.414567:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017515.414572:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017515.414580:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332812800/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017515.414583:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332812416:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017515.414589:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017515.414592:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017515.414594:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017515.414598:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1777 00000100:00080000:3.0:1734017515.414601:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017515.414604:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017515.414608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017515.414616:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332812928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017515.414624:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017515.414627:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017515.414628:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017515.414632:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1777 00000100:00080000:3.0:1734017515.414644:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734017515.414647:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017515.414648:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017515.414653:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017515.414660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332813056/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017515.414668:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017515.414670:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017515/real 1734017515] req@ffff8800a73cf100 x1818247332812416/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017515.414672:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017515.414673:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017515.414677:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1777 00000100:00080000:3.0:1734017515.414680:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017515.414683:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017515.414687:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017515.414695:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332813184/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017515.414697:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332812416/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017515.414707:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017515.414711:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017515.414715:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332812416:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017515.414722:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332812544:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017515.414739:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017515.414746:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017515/real 1734017515] req@ffff8800a73cdf80 x1818247332812544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017515.414756:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332812544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017515.414764:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017515.414766:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017515.414770:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332812544:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017515.414776:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332812672:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017515.414789:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017515.414794:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017515/real 1734017515] req@ffff8800a73cf800 x1818247332812672/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017515.414804:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332812672/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017515.414812:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017515.414815:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017515.414818:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332812672:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017515.414844:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332812800:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017515.414857:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017515.414863:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017515/real 1734017515] req@ffff8800a73cfb80 x1818247332812800/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017515.414873:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332812800/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017515.414881:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017515.414883:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017515.414886:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332812800:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017515.414891:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332812928:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017515.414903:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017515.414908:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017515/real 1734017515] req@ffff8800a73cdc00 x1818247332812928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017515.414918:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332812928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017515.414925:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017515.414927:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017515.414930:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332812928:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017515.414936:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332813056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017515.414951:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017515.414957:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017515/real 1734017515] req@ffff8800a73ce680 x1818247332813056/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017515.414967:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332813056/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017515.414974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017515.414976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017515.414979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332813056:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017515.414985:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332813184:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017515.414997:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017515.415002:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017515/real 1734017515] req@ffff8800a73cc700 x1818247332813184/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017515.415012:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332813184/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017515.415019:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017515.415021:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017515.415024:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332813184:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017515.736440:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017516.738465:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017517.740489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017518.742424:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017519.744470:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017520.422366:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017520.422375:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017520.422380:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017520.422390:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1782 00000100:00080000:3.0:1734017520.422399:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017520.422407:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017520.422444:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332813312/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017520.422478:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017520.422482:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017520.422484:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017520.422489:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1782 00000100:00080000:3.0:1734017520.422492:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017520.422496:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017520.422505:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017520.422516:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332813440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017520.422525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017520.422529:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017520.422531:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017520.422534:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1782 00000100:00080000:3.0:1734017520.422538:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017520.422541:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017520.422546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017520.422554:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332813568/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017520.422563:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017520.422566:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017520.422567:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017520.422570:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1782 00000100:00080000:3.0:1734017520.422573:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017520.422576:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017520.422580:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017520.422588:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332813696/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017520.422597:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017520.422600:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1734017520.422601:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332813312:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017520.422602:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017520.422605:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1782 00000100:00080000:3.0:1734017520.422608:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017520.422611:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017520.422615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017520.422623:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332813824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017520.422637:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017520.422640:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017520.422642:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017520.422646:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1782 00000100:00080000:3.0:1734017520.422649:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017520.422652:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017520.422656:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017520.422663:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332813952/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734017520.422667:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017520.422671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017520.422674:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017520.422675:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017520.422679:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1782 00000100:00100000:1.0:1734017520.422689:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017520/real 1734017520] req@ffff8800a73cc380 x1818247332813312/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017520.422691:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017520.422694:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017520.422699:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017520.422708:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332814080/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017520.422717:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332813312/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017520.422727:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017520.422730:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017520.422734:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332813312:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017520.422741:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332813440:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017520.422759:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017520.422766:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017520/real 1734017520] req@ffff8800a73cca80 x1818247332813440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017520.422777:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332813440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017520.422784:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017520.422787:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017520.422790:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332813440:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017520.422796:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332813568:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017520.422809:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017520.422815:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017520/real 1734017520] req@ffff8800a73cf480 x1818247332813568/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017520.422824:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332813568/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017520.422832:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017520.422835:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017520.422838:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332813568:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017520.422862:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332813696:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017520.422875:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017520.422881:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017520/real 1734017520] req@ffff8800a73cd880 x1818247332813696/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017520.422891:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332813696/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017520.422899:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017520.422901:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017520.422904:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332813696:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017520.422909:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332813824:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017520.422921:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017520.422926:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017520/real 1734017520] req@ffff8800a73cea00 x1818247332813824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017520.422936:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332813824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017520.422943:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017520.422945:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017520.422949:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332813824:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017520.422955:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332813952:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017520.422970:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017520.422976:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017520/real 1734017520] req@ffff8800a73ced80 x1818247332813952/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017520.422986:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332813952/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017520.422993:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017520.422995:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017520.422998:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332813952:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017520.423003:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332814080:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017520.423015:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017520.423020:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017520/real 1734017520] req@ffff8800a73cce00 x1818247332814080/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017520.423030:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332814080/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017520.423037:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017520.423039:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017520.423042:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332814080:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017520.746459:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017521.748488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017522.750466:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017523.752448:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017524.754458:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017525.430327:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017525.430336:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017525.430341:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017525.430351:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1787 00000100:00080000:3.0:1734017525.430360:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017525.430368:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017525.430404:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332814208/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017525.430438:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017525.430442:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017525.430444:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017525.430449:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1787 00000100:00080000:3.0:1734017525.430452:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017525.430456:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017525.430466:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017525.430477:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332814336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017525.430487:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017525.430490:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017525.430492:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017525.430496:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1787 00000100:00080000:3.0:1734017525.430499:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017525.430502:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017525.430506:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017525.430514:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332814464/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017525.430523:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017525.430527:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017525.430528:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017525.430532:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1787 00000100:00080000:3.0:1734017525.430535:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017525.430538:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017525.430542:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017525.430550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332814592/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017525.430553:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332814208:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017525.430559:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017525.430562:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017525.430563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017525.430567:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1787 00000100:00080000:3.0:1734017525.430569:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017525.430573:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017525.430577:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017525.430584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332814720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017525.430592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017525.430595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017525.430596:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017525.430600:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1787 00000100:00080000:3.0:1734017525.430603:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017525.430606:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017525.430611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734017525.430618:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017525.430619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332814848/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017525.430627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017525.430630:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017525.430632:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017525.430635:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1787 00000100:00080000:3.0:1734017525.430637:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734017525.430640:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017525/real 1734017525] req@ffff8800a73cd500 x1818247332814208/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017525.430641:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017525.430645:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017525.430652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332814976/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017525.430667:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332814208/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017525.430678:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017525.430681:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017525.430686:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332814208:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017525.430693:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332814336:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017525.430710:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017525.430717:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017525/real 1734017525] req@ffff8800a73cce00 x1818247332814336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017525.430728:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332814336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017525.430735:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017525.430738:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017525.430741:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332814336:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017525.430747:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332814464:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017525.430761:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017525.430766:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017525/real 1734017525] req@ffff8800a73ced80 x1818247332814464/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017525.430776:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332814464/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017525.430784:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017525.430786:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017525.430790:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332814464:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017525.430814:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332814592:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017525.430826:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017525.430832:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017525/real 1734017525] req@ffff8800a73cea00 x1818247332814592/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017525.430843:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332814592/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017525.430850:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017525.430852:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017525.430855:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332814592:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017525.430861:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332814720:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017525.430872:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017525.430878:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017525/real 1734017525] req@ffff8800a73cd880 x1818247332814720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017525.430887:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332814720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017525.430895:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017525.430897:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017525.430900:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332814720:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017525.430906:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332814848:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017525.430921:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017525.430927:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017525/real 1734017525] req@ffff8800a73cf480 x1818247332814848/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017525.430937:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332814848/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017525.430944:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017525.430946:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017525.430949:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332814848:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017525.430954:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332814976:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017525.430966:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017525.430971:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017525/real 1734017525] req@ffff8800a73cca80 x1818247332814976/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017525.430981:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332814976/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017525.430988:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017525.430990:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017525.430993:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332814976:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017525.756493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017526.758476:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017527.760493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017528.762492:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017529.764490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017530.438364:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017530.438373:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017530.438377:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017530.438388:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1792 00000100:00080000:3.0:1734017530.438398:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017530.438405:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017530.438441:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332815104/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017530.438477:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017530.438484:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017530.438487:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017530.438491:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1792 00000100:00080000:3.0:1734017530.438494:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017530.438499:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017530.438509:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017530.438519:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332815232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017530.438529:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017530.438532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017530.438534:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017530.438538:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1792 00000100:00080000:3.0:1734017530.438540:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017530.438544:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017530.438549:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017530.438557:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332815360/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017530.438565:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017530.438568:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017530.438570:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017530.438573:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1792 00000100:00080000:3.0:1734017530.438576:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017530.438579:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017530.438584:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017530.438592:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332815488/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734017530.438595:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332815104:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017530.438601:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017530.438605:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017530.438606:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017530.438609:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1792 00000100:00080000:3.0:1734017530.438612:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017530.438615:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017530.438620:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017530.438627:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332815616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017530.438635:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017530.438638:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017530.438640:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017530.438644:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1792 00000100:00080000:3.0:1734017530.438656:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017530.438660:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734017530.438661:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017530.438664:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017530.438672:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332815744/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017530.438681:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734017530.438683:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017530/real 1734017530] req@ffff8800a73cc380 x1818247332815104/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017530.438684:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017530.438686:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017530.438690:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1792 00000100:00080000:3.0:1734017530.438693:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017530.438696:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017530.438701:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734017530.438709:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332815104/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017530.438710:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332815872/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734017530.438720:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017530.438724:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017530.438728:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332815104:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734017530.438736:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332815232:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017530.438754:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017530.438761:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017530/real 1734017530] req@ffff8800a73cc700 x1818247332815232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017530.438772:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332815232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017530.438779:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017530.438782:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017530.438785:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332815232:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017530.438791:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332815360:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017530.438804:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017530.438809:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017530/real 1734017530] req@ffff8800a73ce680 x1818247332815360/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017530.438819:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332815360/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017530.438826:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017530.438830:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017530.438833:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332815360:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017530.438857:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332815488:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017530.438870:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017530.438876:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017530/real 1734017530] req@ffff8800a73cdc00 x1818247332815488/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017530.438886:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332815488/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017530.438894:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017530.438896:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017530.438899:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332815488:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017530.438903:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332815616:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734017530.438915:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017530.438921:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017530/real 1734017530] req@ffff8800a73cfb80 x1818247332815616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017530.438930:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332815616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017530.438937:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017530.438939:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017530.438943:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332815616:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734017530.438948:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332815744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017530.438963:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017530.438969:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017530/real 1734017530] req@ffff8800a73cf800 x1818247332815744/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017530.438979:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332815744/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017530.439034:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017530.439036:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017530.439040:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332815744:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734017530.439045:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332815872:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734017530.439061:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734017530.439069:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017530/real 1734017530] req@ffff8800a73cdf80 x1818247332815872/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734017530.439079:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332815872/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734017530.439087:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734017530.439089:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734017530.439092:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332815872:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017530.766473:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017531.768466:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017532.770452:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017533.772449:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017534.774480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017535.446383:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017535.446396:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017535.446402:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017535.446414:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1797 00000100:00080000:3.0:1734017535.446426:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017535.446436:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017535.446479:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332816000/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017535.446511:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017535.446518:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017535.446521:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017535.446527:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1797 00000100:00100000:0.0:1734017535.446543:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332816000:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017535.446548:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017535.446555:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017535.446569:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017535.446591:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332816128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017535.446610:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017535.446616:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017535.446618:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017535.446624:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1797 00000100:00080000:3.0:1734017535.446629:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017535.446635:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017535.446651:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734017535.446656:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332816128:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734017535.446667:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332816256/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017535.446682:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017535.446687:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017535.446690:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017535.446695:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1797 00000100:00100000:0.0:1734017535.446696:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332816256:192.168.201.152@tcp:8:kworker.0 00000100:00080000:3.0:1734017535.446700:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017535.446705:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017535.446710:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017535.446721:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332816384/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017535.446737:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017535.446740:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017535.446742:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017535.446746:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1797 00000100:00080000:3.0:1734017535.446749:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017535.446753:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017535.446758:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017535.446765:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332816384:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734017535.446767:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332816512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017535.446776:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017535.446779:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017535.446780:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017535.446784:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1797 00000100:00080000:3.0:1734017535.446787:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017535.446791:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017535.446795:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017535.446802:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332816512:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734017535.446803:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332816640/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017535.446811:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017535.446815:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017535.446816:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017535.446820:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1797 00000100:00080000:3.0:1734017535.446822:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017535.446826:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017535.446830:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017535.446833:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332816640:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734017535.446838:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332816768/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017535.446861:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332816768:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017535.776491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017536.778466:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017537.780485:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1734017538.452511:0:1957:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:1.0:1734017538.452559:0:1957:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1734017538.452578:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017538.452611:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017538.452619:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017538.452626:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017538.452632:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017538.452638:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017538.452644:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:0.0:1734017538.452710:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017535/real 1734017538] req@ffff8800a73cc700 x1818247332816768/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017538.452748:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332816768/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017538.452759:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017538.452763:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017538.452768:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332816768:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017538.452776:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017535/real 1734017538] req@ffff8800a73ce680 x1818247332816640/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017538.452787:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332816640/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017538.452795:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017538.452797:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017538.452801:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332816640:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017538.452807:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017535/real 1734017538] req@ffff8800a73cdc00 x1818247332816512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017538.452818:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332816512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017538.452826:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017538.452828:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017538.452832:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332816512:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017538.452838:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017535/real 1734017538] req@ffff8800a73cfb80 x1818247332816384/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017538.452911:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332816384/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017538.452919:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017538.452922:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017538.452925:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332816384:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017538.452931:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017535/real 1734017538] req@ffff8800a73cf800 x1818247332816256/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017538.452945:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332816256/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017538.452957:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017538.452961:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017538.452967:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332816256:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017538.452977:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017535/real 1734017538] req@ffff8800a73cdf80 x1818247332816128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017538.452993:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332816128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017538.453006:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017538.453011:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017538.453016:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332816128:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017538.453025:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017535/real 1734017538] req@ffff8800a73cf100 x1818247332816000/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017538.453055:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332816000/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017538.453067:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017538.453070:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017538.453076:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332816000:192.168.201.152@tcp:250:kworker.0 00010000:00010000:2.0:1734017538.782519:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017539.784458:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017540.454367:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017540.454377:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017540.454382:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017540.454392:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1802 00000100:00080000:3.0:1734017540.454402:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017540.454410:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017540.454447:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332816896/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017540.454480:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017540.454485:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017540.454487:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017540.454492:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1802 00000100:00080000:3.0:1734017540.454495:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017540.454499:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017540.454508:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017540.454519:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332817024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017540.454529:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017540.454532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017540.454534:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017540.454537:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1802 00000100:00080000:3.0:1734017540.454541:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017540.454545:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017540.454550:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017540.454558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332817152/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017540.454566:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017540.454569:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017540.454571:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017540.454574:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1802 00000100:00080000:3.0:1734017540.454576:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017540.454580:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017540.454584:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017540.454585:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332816896:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017540.454592:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332817280/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017540.454601:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017540.454604:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017540.454605:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017540.454609:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1802 00000100:00080000:3.0:1734017540.454611:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017540.454615:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017540.454619:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017540.454627:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332817408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017540.454642:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017540.454645:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017540.454646:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1734017540.454650:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017540.454650:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1802 00000100:00080000:3.0:1734017540.454653:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017540.454656:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017540.454660:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017540.454667:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332817536/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017540.454671:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017540/real 1734017540] req@ffff8800a73cc380 x1818247332816896/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017540.454675:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017540.454678:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017540.454680:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017540.454683:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1802 00000100:00080000:3.0:1734017540.454686:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017540.454690:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332816896/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017540.454690:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017540.454694:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1734017540.454701:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1734017540.454703:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332817664/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017540.454704:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017540.454709:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332816896:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017540.454717:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332817024:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017540.454734:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017540.454740:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017540/real 1734017540] req@ffff8800a73cca80 x1818247332817024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017540.454752:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332817024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017540.454760:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017540.454763:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017540.454766:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332817024:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017540.454773:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332817152:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017540.454786:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017540.454792:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017540/real 1734017540] req@ffff8800a73cf480 x1818247332817152/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017540.454802:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332817152/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017540.454809:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017540.454812:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017540.454858:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332817152:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017540.454884:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332817280:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017540.454901:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017540.454908:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017540/real 1734017540] req@ffff8800a73cd880 x1818247332817280/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017540.454919:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332817280/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017540.454927:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017540.454930:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017540.454933:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332817280:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017540.454938:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332817408:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017540.454954:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017540.454960:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017540/real 1734017540] req@ffff8800a73cea00 x1818247332817408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017540.454971:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332817408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017540.454979:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017540.454981:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017540.454984:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332817408:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017540.454989:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332817536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017540.455001:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017540.455007:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017540/real 1734017540] req@ffff8800a73ced80 x1818247332817536/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017540.455016:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332817536/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017540.455023:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017540.455025:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017540.455028:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332817536:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017540.455034:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332817664:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017540.455046:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017540.455052:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017540/real 1734017540] req@ffff8800a73cce00 x1818247332817664/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017540.455069:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332817664/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017540.455076:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017540.455078:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017540.455081:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332817664:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017540.786508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017541.788465:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017542.790476:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017543.792454:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017544.794457:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017545.462373:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017545.462382:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017545.462387:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017545.462398:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1808 00000100:00080000:3.0:1734017545.462408:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017545.462416:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017545.462451:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332817792/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017545.462484:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017545.462489:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017545.462491:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017545.462495:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1808 00000100:00080000:3.0:1734017545.462498:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017545.462502:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017545.462512:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017545.462524:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332817920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017545.462534:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017545.462537:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017545.462539:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017545.462542:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1808 00000100:00080000:3.0:1734017545.462545:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017545.462548:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017545.462553:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017545.462561:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332818048/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017545.462569:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017545.462574:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017545.462575:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017545.462579:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1808 00000100:00080000:3.0:1734017545.462581:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017545.462585:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017545.462589:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017545.462590:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332817792:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017545.462596:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332818176/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017545.462605:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017545.462608:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017545.462610:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017545.462613:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1808 00000100:00080000:3.0:1734017545.462615:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017545.462619:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017545.462623:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017545.462630:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332818304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017545.462638:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017545.462641:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017545.462642:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017545.462646:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1808 00000100:00080000:3.0:1734017545.462649:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017545.462652:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734017545.462654:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017545.462657:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017545.462665:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332818432/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017545.462673:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017545.462676:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017545/real 1734017545] req@ffff8800a73cd500 x1818247332817792/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017545.462676:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017545.462678:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017545.462681:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1808 00000100:00080000:3.0:1734017545.462684:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017545.462687:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017545.462691:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017545.462695:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332817792/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017545.462699:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332818560/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017545.462706:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017545.462709:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017545.462714:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332817792:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017545.462722:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332817920:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017545.462773:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017545.462781:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017545/real 1734017545] req@ffff8800a73cce00 x1818247332817920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017545.462793:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332817920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017545.462801:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017545.462804:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017545.462808:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332817920:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017545.462815:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332818048:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017545.462829:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017545.462835:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017545/real 1734017545] req@ffff8800a73ced80 x1818247332818048/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017545.462845:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332818048/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017545.462853:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017545.462855:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017545.462859:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332818048:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017545.462884:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332818176:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017545.462896:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017545.462902:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017545/real 1734017545] req@ffff8800a73cea00 x1818247332818176/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017545.462912:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332818176/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017545.462920:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017545.462922:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017545.462925:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332818176:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017545.462931:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332818304:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017545.462946:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017545.462952:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017545/real 1734017545] req@ffff8800a73cd880 x1818247332818304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017545.462962:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332818304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017545.462970:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017545.462972:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017545.462975:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332818304:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017545.462981:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332818432:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017545.462992:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017545.462998:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017545/real 1734017545] req@ffff8800a73cf480 x1818247332818432/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017545.463008:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332818432/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017545.463015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017545.463017:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017545.463020:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332818432:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017545.463025:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332818560:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017545.463037:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017545.463043:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017545/real 1734017545] req@ffff8800a73cca80 x1818247332818560/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017545.463060:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332818560/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017545.463068:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017545.463070:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017545.463073:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332818560:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017545.796469:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017546.798497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017547.800453:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017548.802515:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017549.804501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017550.470330:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017550.470339:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017550.470344:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017550.470355:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1813 00000100:00080000:3.0:1734017550.470365:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017550.470373:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017550.470409:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332818688/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017550.470442:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017550.470449:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017550.470451:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017550.470455:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1813 00000100:00080000:3.0:1734017550.470459:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017550.470463:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017550.470472:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017550.470483:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332818816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017550.470493:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017550.470497:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017550.470498:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017550.470502:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1813 00000100:00080000:3.0:1734017550.470505:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017550.470508:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017550.470513:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017550.470521:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332818944/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017550.470529:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017550.470532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017550.470534:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017550.470537:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1813 00000100:00080000:3.0:1734017550.470539:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017550.470543:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017550.470547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017550.470554:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332818688:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017550.470556:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332819072/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017550.470565:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017550.470568:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017550.470570:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017550.470573:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1813 00000100:00080000:3.0:1734017550.470576:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017550.470579:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017550.470583:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017550.470591:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332819200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017550.470599:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017550.470602:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017550.470603:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017550.470607:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1813 00000100:00080000:3.0:1734017550.470611:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017550.470615:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017550.470619:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017550.470621:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017550.470627:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332819328/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017550.470636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017550.470639:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734017550.470641:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017550/real 1734017550] req@ffff8800a73cc380 x1818247332818688/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017550.470641:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017550.470678:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1813 00000100:00080000:3.0:1734017550.470682:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017550.470685:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017550.470691:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017550.470701:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332819456/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017550.470705:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332818688/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017550.470716:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017550.470720:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017550.470724:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332818688:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017550.470732:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332818816:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017550.470752:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017550.470760:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017550/real 1734017550] req@ffff8800a73cf100 x1818247332818816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017550.470771:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332818816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017550.470779:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017550.470783:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017550.470786:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332818816:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017550.470793:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332818944:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017550.470808:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017550.470813:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017550/real 1734017550] req@ffff8800a73cdf80 x1818247332818944/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017550.470824:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332818944/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017550.470832:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017550.470834:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017550.470837:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332818944:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017550.470863:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332819072:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017550.470875:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017550.470881:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017550/real 1734017550] req@ffff8800a73cf800 x1818247332819072/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017550.470892:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332819072/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017550.470899:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017550.470901:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017550.470904:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332819072:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017550.470909:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332819200:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017550.470925:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017550.470931:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017550/real 1734017550] req@ffff8800a73cfb80 x1818247332819200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017550.470940:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332819200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017550.470948:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017550.470950:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017550.470953:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332819200:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017550.470958:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332819328:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017550.470969:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017550.470975:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017550/real 1734017550] req@ffff8800a73cdc00 x1818247332819328/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017550.470985:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332819328/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017550.470992:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017550.470994:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017550.470997:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332819328:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017550.471002:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332819456:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017550.471014:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017550.471020:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017550/real 1734017550] req@ffff8800a73ce680 x1818247332819456/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017550.471036:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332819456/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017550.471044:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017550.471046:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017550.471049:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332819456:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017550.806486:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017551.808457:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017552.810485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017553.812492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017554.814470:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017555.478372:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017555.478381:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017555.478386:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017555.478397:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1818 00000100:00080000:3.0:1734017555.478406:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017555.478414:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017555.478451:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332819584/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017555.478484:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017555.478489:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017555.478491:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017555.478495:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1818 00000100:00080000:3.0:1734017555.478498:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017555.478502:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017555.478512:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017555.478525:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332819712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017555.478536:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017555.478539:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017555.478541:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017555.478544:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1818 00000100:00080000:3.0:1734017555.478547:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017555.478550:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017555.478555:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017555.478605:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332819840/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017555.478615:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017555.478618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734017555.478620:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332819584:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017555.478620:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017555.478624:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1818 00000100:00080000:3.0:1734017555.478626:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017555.478630:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017555.478634:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017555.478642:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332819968/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017555.478651:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017555.478654:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017555.478656:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017555.478659:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1818 00000100:00080000:3.0:1734017555.478664:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017555.478667:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017555.478672:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017555.478680:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332820096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734017555.478685:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017555.478688:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017555.478691:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017555.478692:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017555.478696:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1818 00000100:00100000:0.0:1734017555.478706:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017555/real 1734017555] req@ffff8800a73cc700 x1818247332819584/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017555.478708:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017555.478712:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017555.478717:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017555.478724:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332820224/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017555.478725:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332819584/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017555.478732:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017555.478735:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734017555.478736:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017555.478737:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017555.478739:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734017555.478741:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1818 00000100:00080000:3.0:1734017555.478743:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017555.478744:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332819584:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017555.478747:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017555.478751:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332819712:192.168.201.152@tcp:38:kworker.0 00000008:00100000:3.0:1734017555.478751:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017555.478760:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332820352/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734017555.478769:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017555.478776:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017555/real 1734017555] req@ffff8800a73ce680 x1818247332819712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017555.478787:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332819712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017555.478795:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017555.478798:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017555.478802:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332819712:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017555.478825:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332819840:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017555.478839:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017555.478845:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017555/real 1734017555] req@ffff8800a73cdc00 x1818247332819840/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017555.478855:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332819840/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017555.478863:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017555.478865:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017555.478868:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332819840:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017555.478874:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332819968:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017555.478885:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017555.478891:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017555/real 1734017555] req@ffff8800a73cfb80 x1818247332819968/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017555.478900:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332819968/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017555.478908:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017555.478910:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017555.478913:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332819968:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017555.478919:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332820096:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017555.478934:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017555.478940:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017555/real 1734017555] req@ffff8800a73cf800 x1818247332820096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017555.478950:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332820096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017555.478957:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017555.478959:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017555.478962:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332820096:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017555.478968:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332820224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017555.478979:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017555.478985:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017555/real 1734017555] req@ffff8800a73cdf80 x1818247332820224/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017555.478994:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332820224/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017555.479001:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017555.479003:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017555.479006:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332820224:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017555.479012:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332820352:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017555.479024:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017555.479030:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017555/real 1734017555] req@ffff8800a73cf100 x1818247332820352/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017555.479047:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332820352/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017555.479055:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017555.479057:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017555.479060:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332820352:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017555.816485:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017556.818475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017557.820560:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017558.822489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017559.824516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017560.486364:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017560.486374:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017560.486379:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017560.486390:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1823 00000100:00080000:3.0:1734017560.486400:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017560.486407:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017560.486442:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332820480/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017560.486502:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017560.486509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017560.486511:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017560.486518:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1823 00000100:00080000:3.0:1734017560.486523:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017560.486530:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017560.486542:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017560.486562:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332820608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017560.486564:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332820480:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017560.486579:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017560.486583:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017560.486586:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017560.486591:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1823 00000100:00080000:3.0:1734017560.486596:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017560.486601:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017560.486608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017560.486619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332820736/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017560.486631:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1734017560.486635:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017560.486636:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017560.486639:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017560.486643:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1823 00000100:00080000:3.0:1734017560.486647:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017560.486652:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017560.486658:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017560/real 1734017560] req@ffff8800a73cc380 x1818247332820480/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017560.486658:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017560.486671:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332820864/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017560.486682:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332820480/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017560.486683:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017560.486689:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017560.486691:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017560.486696:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1823 00000100:00080000:0.0:1734017560.486697:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017560.486700:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017560.486702:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734017560.486705:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017560.486708:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332820480:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017560.486712:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017560.486723:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332820992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017560.486734:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332820608:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017560.486746:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017560.486751:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017560.486754:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1734017560.486759:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017560.486760:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1823 00000100:00080000:3.0:1734017560.486764:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017560.486769:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017560.486770:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017560/real 1734017560] req@ffff8800a73cca80 x1818247332820608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017560.486777:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017560.486788:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332820608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017560.486790:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332821120/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017560.486802:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017560.486804:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017560.486806:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734017560.486809:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734017560.486812:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332820608:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017560.486812:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017560.486817:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1823 00000100:00100000:0.0:1734017560.486821:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332820736:192.168.201.152@tcp:8:kworker.0 00000100:00080000:3.0:1734017560.486821:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017560.486826:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017560.486832:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017560.486842:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017560.486846:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332821248/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017560.486853:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017560/real 1734017560] req@ffff8800a73cf480 x1818247332820736/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017560.486870:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332820736/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017560.486882:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017560.486886:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017560.486891:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332820736:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017560.486900:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332820864:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017560.486926:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017560.486938:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017560/real 1734017560] req@ffff8800a73cd880 x1818247332820864/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017560.486954:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332820864/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017560.486963:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017560.486965:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017560.486968:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332820864:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017560.486986:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332820992:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017560.487006:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017560.487012:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017560/real 1734017560] req@ffff8800a73cea00 x1818247332820992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017560.487023:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332820992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017560.487030:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017560.487032:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017560.487035:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332820992:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017560.487040:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332821120:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017560.487052:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017560.487058:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017560/real 1734017560] req@ffff8800a73ced80 x1818247332821120/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017560.487069:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332821120/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017560.487076:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017560.487078:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017560.487081:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332821120:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017560.487085:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332821248:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017560.487097:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017560.487103:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017560/real 1734017560] req@ffff8800a73cce00 x1818247332821248/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017560.487119:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332821248/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017560.487127:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017560.487129:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017560.487132:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332821248:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017560.826449:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017561.828518:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017562.830518:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017563.832494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017564.834471:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017565.494334:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017565.494343:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017565.494347:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017565.494358:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1828 00000100:00080000:3.0:1734017565.494367:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017565.494375:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017565.494438:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332821376/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017565.494471:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017565.494477:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017565.494480:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017565.494486:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1828 00000100:00080000:3.0:1734017565.494491:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017565.494498:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017565.494510:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017565.494529:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332821504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017565.494538:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332821376:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017565.494546:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017565.494550:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017565.494552:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017565.494557:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1828 00000100:00080000:3.0:1734017565.494560:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017565.494565:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017565.494572:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017565.494585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332821632/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017565.494597:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017565.494602:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017565.494605:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1734017565.494607:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017565.494610:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1828 00000100:00080000:3.0:1734017565.494614:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017565.494618:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017565.494624:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017565.494631:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017565/real 1734017565] req@ffff8800a73cd500 x1818247332821376/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017565.494636:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332821760/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017565.494649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017565.494655:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332821376/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017565.494655:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017565.494657:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017565.494663:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1828 00000100:00080000:3.0:1734017565.494666:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017565.494669:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017565.494671:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:0.0:1734017565.494674:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000008:00100000:3.0:1734017565.494678:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017565.494681:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332821376:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017565.494690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332821888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017565.494702:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017565.494707:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332821504:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017565.494707:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017565.494710:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017565.494715:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1828 00000100:00080000:3.0:1734017565.494719:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017565.494724:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017565.494731:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017565.494732:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017565.494744:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017565/real 1734017565] req@ffff8800a73cce00 x1818247332821504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017565.494744:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332822016/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017565.494759:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017565.494761:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332821504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017565.494764:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017565.494767:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017565.494772:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1828 00000100:00080000:0.0:1734017565.494774:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017565.494776:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017565.494778:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734017565.494781:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017565.494784:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332821504:192.168.201.152@tcp:38:kworker.0 00000008:00100000:3.0:1734017565.494788:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017565.494793:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332821632:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734017565.494799:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332822144/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734017565.494814:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017565.494824:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017565/real 1734017565] req@ffff8800a73ced80 x1818247332821632/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017565.494841:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332821632/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017565.494854:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017565.494858:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017565.494863:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332821632:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017565.494872:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332821760:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017565.494897:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017565.494906:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017565/real 1734017565] req@ffff8800a73cea00 x1818247332821760/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017565.494917:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332821760/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017565.494925:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017565.494927:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017565.494931:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332821760:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017565.494936:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332821888:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017565.494953:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017565.494959:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017565/real 1734017565] req@ffff8800a73cd880 x1818247332821888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017565.494970:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332821888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017565.494977:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017565.494979:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017565.494982:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332821888:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017565.494999:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332822016:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017565.495011:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017565.495017:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017565/real 1734017565] req@ffff8800a73cf480 x1818247332822016/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017565.495027:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332822016/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017565.495034:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017565.495036:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017565.495040:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332822016:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017565.495045:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332822144:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017565.495056:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017565.495062:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017565/real 1734017565] req@ffff8800a73cca80 x1818247332822144/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017565.495078:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332822144/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017565.495086:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017565.495088:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017565.495091:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332822144:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017565.836457:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017566.838515:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017567.840471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017568.842500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017569.844489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017570.502381:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017570.502390:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017570.502395:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017570.502405:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1833 00000100:00080000:3.0:1734017570.502415:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017570.502423:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017570.502460:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332822272/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017570.502493:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017570.502500:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017570.502502:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017570.502506:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1833 00000100:00080000:3.0:1734017570.502510:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017570.502514:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017570.502524:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017570.502534:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332822400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017570.502544:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017570.502547:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017570.502549:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017570.502553:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1833 00000100:00080000:3.0:1734017570.502556:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017570.502559:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017570.502564:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017570.502572:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332822528/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017570.502580:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017570.502583:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017570.502585:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017570.502588:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1833 00000100:00080000:3.0:1734017570.502591:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017570.502594:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017570.502599:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017570.502607:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332822656/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017570.502615:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017570.502619:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017570.502620:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017570.502623:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1833 00000100:00080000:3.0:1734017570.502625:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017570.502629:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017570.502633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017570.502640:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332822784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017570.502649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017570.502652:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017570.502653:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017570.502657:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332822272:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017570.502657:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1833 00000100:00080000:3.0:1734017570.502660:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017570.502663:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017570.502668:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017570.502675:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332822912/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017570.502683:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017570.502686:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017570.502689:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017570.502693:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1833 00000100:00080000:3.0:1734017570.502695:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017570.502699:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017570.502703:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017570.502710:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332823040/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734017570.502751:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017570.502780:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017570/real 1734017570] req@ffff8800a73cc380 x1818247332822272/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017570.502802:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332822272/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017570.502816:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017570.502820:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017570.502825:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332822272:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017570.502833:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332822400:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017570.502852:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017570.502859:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017570/real 1734017570] req@ffff8800a73cf100 x1818247332822400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017570.502870:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332822400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017570.502878:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017570.502883:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017570.502886:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332822400:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017570.502894:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332822528:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017570.502910:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017570.502916:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017570/real 1734017570] req@ffff8800a73cdf80 x1818247332822528/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017570.502926:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332822528/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017570.502934:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017570.502937:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017570.502940:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332822528:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017570.502946:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332822656:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017570.502959:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017570.502965:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017570/real 1734017570] req@ffff8800a73cf800 x1818247332822656/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017570.502975:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332822656/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017570.502982:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017570.502985:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017570.502987:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332822656:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017570.503020:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332822784:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017570.503037:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017570.503043:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017570/real 1734017570] req@ffff8800a73cfb80 x1818247332822784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017570.503054:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332822784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017570.503062:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017570.503065:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017570.503067:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332822784:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017570.503073:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332822912:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017570.503086:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017570.503092:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017570/real 1734017570] req@ffff8800a73cdc00 x1818247332822912/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017570.503101:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332822912/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017570.503108:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017570.503111:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017570.503114:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332822912:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017570.503119:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332823040:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017570.503132:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017570.503139:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017570/real 1734017570] req@ffff8800a73ce680 x1818247332823040/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017570.503156:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332823040/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017570.503165:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017570.503167:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017570.503170:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332823040:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017570.846465:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017571.848508:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017572.850513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017573.852496:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017574.854432:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017575.510315:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017575.510324:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017575.510328:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017575.510338:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1838 00000100:00080000:3.0:1734017575.510347:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017575.510355:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017575.510392:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332823168/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017575.510436:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017575.510443:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017575.510446:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017575.510452:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1838 00000100:00080000:3.0:1734017575.510458:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017575.510464:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017575.510476:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017575.510495:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332823296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017575.510510:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017575.510515:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017575.510517:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017575.510522:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1838 00000100:00080000:3.0:1734017575.510526:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017575.510531:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017575.510538:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734017575.510539:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332823168:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017575.510552:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332823424/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017575.510566:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017575.510570:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017575.510573:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017575.510578:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1838 00000100:00080000:3.0:1734017575.510582:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017575.510588:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017575.510594:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017575.510605:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017575.510607:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332823552/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017575.510619:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017575.510624:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1734017575.510627:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017575/real 1734017575] req@ffff8800a73cc700 x1818247332823168/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017575.510627:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017575.510632:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1838 00000100:00080000:3.0:1734017575.510637:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017575.510642:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017575.510646:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332823168/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017575.510648:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1734017575.510657:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017575.510660:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017575.510662:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332823680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017575.510665:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332823168:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017575.510672:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332823296:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017575.510674:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017575.510678:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017575.510681:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017575.510687:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1838 00000800:00000100:0.0:1734017575.510689:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017575.510692:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017575.510696:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017575/real 1734017575] req@ffff8800a73ce680 x1818247332823296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017575.510698:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017575.510704:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017575.510708:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332823296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017575.510716:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1734017575.510718:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332823808/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017575.510719:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017575.510723:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332823296:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017575.510731:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017575.510735:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017575.510738:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017575.510744:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1838 00000100:00100000:0.0:1734017575.510746:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332823424:192.168.201.152@tcp:8:kworker.0 00000100:00080000:3.0:1734017575.510748:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017575.510753:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017575.510759:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017575.510761:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017575.510767:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017575/real 1734017575] req@ffff8800a73cdc00 x1818247332823424/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017575.510770:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332823936/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017575.510778:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332823424/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017575.510786:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017575.510788:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017575.510791:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332823424:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017575.510797:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332823552:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017575.510808:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017575.510814:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017575/real 1734017575] req@ffff8800a73cfb80 x1818247332823552/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017575.510824:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332823552/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017575.510832:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017575.510834:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017575.510837:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332823552:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017575.510843:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332823680:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017575.510859:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017575.510864:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017575/real 1734017575] req@ffff8800a73cf800 x1818247332823680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017575.510875:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332823680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017575.510882:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017575.510885:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017575.510888:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332823680:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017575.510893:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332823808:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017575.510904:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017575.510910:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017575/real 1734017575] req@ffff8800a73cdf80 x1818247332823808/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017575.510920:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332823808/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017575.510928:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017575.510930:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017575.510933:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332823808:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017575.510948:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332823936:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017575.510961:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017575.510967:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017575/real 1734017575] req@ffff8800a73cf100 x1818247332823936/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017575.510983:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332823936/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017575.510991:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017575.510993:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017575.510996:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332823936:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017575.856485:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017576.858448:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017577.860496:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017578.862495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017579.864485:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017580.518340:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017580.518350:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017580.518354:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017580.518366:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1843 00000100:00080000:3.0:1734017580.518375:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017580.518383:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017580.518418:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332824064/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017580.518451:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017580.518456:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017580.518457:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017580.518462:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1843 00000100:00080000:3.0:1734017580.518465:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017580.518469:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017580.518478:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017580.518489:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332824192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017580.518499:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017580.518502:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017580.518504:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017580.518508:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1843 00000100:00080000:3.0:1734017580.518511:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017580.518515:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017580.518520:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017580.518528:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332824320/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017580.518537:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017580.518540:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017580.518541:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017580.518544:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1843 00000100:00080000:3.0:1734017580.518547:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017580.518550:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017580.518554:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332824064:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017580.518555:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017580.518562:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332824448/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017580.518571:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017580.518574:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017580.518575:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017580.518579:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1843 00000100:00080000:3.0:1734017580.518581:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017580.518584:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017580.518588:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017580.518596:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332824576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017580.518611:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017580.518614:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017580.518616:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017580.518619:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1843 00000800:00000100:0.0:1734017580.518620:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017580.518622:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017580.518625:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017580.518629:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017580.518636:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332824704/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017580.518642:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017580/real 1734017580] req@ffff8800a73cc380 x1818247332824064/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017580.518644:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017580.518647:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017580.518649:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017580.518652:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1843 00000100:00080000:3.0:1734017580.518655:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017580.518658:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017580.518661:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332824064/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017580.518662:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017580.518669:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332824832/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017580.518672:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017580.518675:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017580.518680:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332824064:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017580.518688:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332824192:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017580.518705:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017580.518711:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017580/real 1734017580] req@ffff8800a73cca80 x1818247332824192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017580.518722:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332824192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017580.518730:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017580.518734:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017580.518737:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332824192:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017580.518744:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332824320:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017580.518757:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017580.518763:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017580/real 1734017580] req@ffff8800a73cf480 x1818247332824320/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017580.518773:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332824320/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017580.518781:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017580.518783:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017580.518786:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332824320:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017580.518810:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332824448:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017580.518822:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017580.518828:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017580/real 1734017580] req@ffff8800a73cd880 x1818247332824448/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017580.518839:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332824448/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017580.518847:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017580.518849:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017580.518852:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332824448:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017580.518857:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332824576:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017580.518873:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017580.518878:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017580/real 1734017580] req@ffff8800a73cea00 x1818247332824576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017580.518889:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332824576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017580.518896:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017580.518898:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017580.518901:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332824576:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017580.518906:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332824704:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017580.518917:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017580.518923:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017580/real 1734017580] req@ffff8800a73ced80 x1818247332824704/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017580.518933:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332824704/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017580.518940:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017580.518943:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017580.518946:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332824704:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017580.518951:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332824832:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017580.518963:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017580.518968:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017580/real 1734017580] req@ffff8800a73cce00 x1818247332824832/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017580.518984:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332824832/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017580.518992:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017580.518994:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017580.518997:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332824832:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017580.866465:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017581.868455:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017582.870498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017583.872486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017584.874415:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017585.526364:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017585.526373:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017585.526378:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017585.526389:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1848 00000100:00080000:3.0:1734017585.526398:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017585.526406:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017585.526442:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332824960/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017585.526475:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017585.526479:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017585.526482:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017585.526486:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1848 00000100:00080000:3.0:1734017585.526489:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017585.526493:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017585.526503:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017585.526516:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332825088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017585.526525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017585.526528:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017585.526530:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017585.526534:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1848 00000100:00080000:3.0:1734017585.526537:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017585.526540:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017585.526545:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017585.526554:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332825216/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017585.526563:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017585.526566:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017585.526568:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017585.526571:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1848 00000100:00080000:3.0:1734017585.526574:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017585.526575:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332824960:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017585.526577:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017585.526582:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017585.526589:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332825344/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017585.526597:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017585.526600:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017585.526602:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017585.526605:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1848 00000100:00080000:3.0:1734017585.526608:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017585.526611:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017585.526615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017585.526623:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332825472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017585.526631:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017585.526634:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017585.526635:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017585.526639:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1848 00000800:00000100:0.0:1734017585.526640:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017585.526642:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017585.526646:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017585.526651:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017585.526659:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332825600/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017585.526662:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017585/real 1734017585] req@ffff8800a73cd500 x1818247332824960/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017585.526667:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017585.526670:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017585.526672:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017585.526675:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1848 00000100:00080000:3.0:1734017585.526678:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017585.526681:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332824960/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017585.526681:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017585.526685:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1734017585.526692:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1734017585.526692:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332825728/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017585.526696:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017585.526700:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332824960:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017585.526708:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332825088:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017585.526725:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017585.526732:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017585/real 1734017585] req@ffff8800a73cce00 x1818247332825088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017585.526743:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332825088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017585.526750:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017585.526754:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017585.526757:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332825088:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017585.526785:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332825216:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017585.526799:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017585.526805:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017585/real 1734017585] req@ffff8800a73ced80 x1818247332825216/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017585.526816:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332825216/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017585.526824:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017585.526826:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017585.526829:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332825216:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017585.526834:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332825344:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017585.526845:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017585.526851:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017585/real 1734017585] req@ffff8800a73cea00 x1818247332825344/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017585.526861:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332825344/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017585.526868:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017585.526870:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017585.526874:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332825344:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017585.526879:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332825472:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017585.526894:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017585.526900:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017585/real 1734017585] req@ffff8800a73cd880 x1818247332825472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017585.526911:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332825472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017585.526918:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017585.526921:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017585.526924:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332825472:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017585.526930:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332825600:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017585.526941:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017585.526947:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017585/real 1734017585] req@ffff8800a73cf480 x1818247332825600/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017585.526956:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332825600/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017585.526963:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017585.526965:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017585.526969:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332825600:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017585.526975:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332825728:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017585.526986:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017585.526992:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017585/real 1734017585] req@ffff8800a73cca80 x1818247332825728/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017585.527008:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332825728/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017585.527016:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017585.527018:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017585.527021:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332825728:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017585.876498:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017586.878488:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017587.880523:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017588.882448:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017589.884495:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017590.534371:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017590.534380:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017590.534384:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017590.534395:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1853 00000100:00080000:3.0:1734017590.534404:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017590.534412:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017590.534447:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332825856/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017590.534481:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017590.534488:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017590.534490:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017590.534495:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1853 00000100:00080000:3.0:1734017590.534498:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017590.534502:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017590.534512:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017590.534522:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332825984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017590.534532:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017590.534536:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017590.534537:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017590.534541:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1853 00000100:00080000:3.0:1734017590.534544:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017590.534548:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017590.534552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017590.534560:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332826112/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017590.534568:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017590.534571:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017590.534573:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017590.534576:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1853 00000100:00080000:3.0:1734017590.534579:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017590.534583:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017590.534588:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332825856:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017590.534588:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017590.534596:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332826240/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017590.534605:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017590.534609:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017590.534610:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017590.534613:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1853 00000100:00080000:3.0:1734017590.534616:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017590.534619:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017590.534623:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017590.534631:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332826368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017590.534639:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017590.534642:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017590.534644:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017590.534647:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1853 00000100:00080000:3.0:1734017590.534650:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017590.534653:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734017590.534654:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017590.534658:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017590.534665:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332826496/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017590.534673:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017590.534675:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017590/real 1734017590] req@ffff8800a73cc380 x1818247332825856/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017590.534676:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017590.534679:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017590.534683:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1853 00000100:00080000:3.0:1734017590.534685:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017590.534689:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017590.534693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017590.534694:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332825856/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017590.534702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332826624/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017590.534704:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017590.534708:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017590.534713:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332825856:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017590.534720:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332825984:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017590.534737:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017590.534744:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017590/real 1734017590] req@ffff8800a73cf100 x1818247332825984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017590.534755:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332825984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017590.534763:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017590.534767:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017590.534770:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332825984:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017590.534777:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332826112:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017590.534790:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017590.534796:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017590/real 1734017590] req@ffff8800a73cdf80 x1818247332826112/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017590.534806:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332826112/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017590.534814:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017590.534816:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017590.534820:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332826112:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017590.534843:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332826240:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017590.534855:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017590.534862:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017590/real 1734017590] req@ffff8800a73cf800 x1818247332826240/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017590.534872:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332826240/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017590.534880:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017590.534882:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017590.534885:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332826240:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017590.534890:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332826368:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017590.534905:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017590.534911:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017590/real 1734017590] req@ffff8800a73cfb80 x1818247332826368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017590.534922:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332826368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017590.534929:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017590.534931:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017590.534934:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332826368:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017590.534940:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332826496:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017590.534952:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017590.534958:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017590/real 1734017590] req@ffff8800a73cdc00 x1818247332826496/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017590.535004:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332826496/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017590.535012:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017590.535014:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017590.535017:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332826496:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017590.535023:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332826624:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017590.535038:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017590.535045:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017590/real 1734017590] req@ffff8800a73ce680 x1818247332826624/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017590.535061:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332826624/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017590.535069:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017590.535071:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017590.535074:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332826624:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017590.886494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017591.888477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017592.890439:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017593.892498:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017594.894472:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017595.542393:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017595.542403:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017595.542408:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017595.542418:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1858 00000100:00080000:3.0:1734017595.542428:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017595.542436:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017595.542473:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332826752/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017595.542509:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017595.542513:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017595.542516:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017595.542520:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1858 00000100:00080000:3.0:1734017595.542523:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017595.542527:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017595.542537:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017595.542549:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332826880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017595.542561:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017595.542564:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017595.542566:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017595.542569:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1858 00000100:00080000:3.0:1734017595.542572:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017595.542575:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017595.542580:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017595.542588:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332827008/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017595.542597:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017595.542600:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017595.542601:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017595.542605:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1858 00000100:00080000:3.0:1734017595.542607:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017595.542611:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017595.542615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017595.542622:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332827136/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017595.542632:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017595.542635:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017595.542637:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017595.542641:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1858 00000100:00080000:3.0:1734017595.542644:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017595.542645:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332826752:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017595.542648:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017595.542652:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017595.542660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332827264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017595.542668:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017595.542671:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017595.542673:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017595.542677:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1858 00000100:00080000:3.0:1734017595.542680:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017595.542683:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017595.542687:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017595.542694:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332827392/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017595.542702:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017595.542705:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017595.542707:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017595.542710:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1858 00000100:00080000:3.0:1734017595.542713:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017595.542716:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017595.542720:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017595.542728:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332827520/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734017595.542737:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017595.542766:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017595/real 1734017595] req@ffff8800a73cc700 x1818247332826752/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017595.542789:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332826752/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017595.542802:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017595.542806:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017595.542811:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332826752:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017595.542819:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332826880:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017595.542838:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017595.542845:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017595/real 1734017595] req@ffff8800a73ce680 x1818247332826880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017595.542857:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332826880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017595.542864:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017595.542869:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017595.542872:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332826880:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017595.542932:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332827008:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017595.542952:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017595.542960:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017595/real 1734017595] req@ffff8800a73cdc00 x1818247332827008/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017595.542972:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332827008/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017595.542980:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017595.542983:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017595.542986:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332827008:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017595.543015:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332827136:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017595.543029:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017595.543035:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017595/real 1734017595] req@ffff8800a73cfb80 x1818247332827136/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017595.543045:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332827136/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017595.543053:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017595.543055:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017595.543059:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332827136:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017595.543064:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332827264:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017595.543081:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017595.543088:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017595/real 1734017595] req@ffff8800a73cf800 x1818247332827264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017595.543098:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332827264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017595.543106:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017595.543108:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017595.543111:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332827264:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017595.543117:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332827392:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017595.543129:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017595.543135:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017595/real 1734017595] req@ffff8800a73cdf80 x1818247332827392/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017595.543145:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332827392/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017595.543153:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017595.543156:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017595.543159:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332827392:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017595.543164:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332827520:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017595.543177:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017595.543183:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017595/real 1734017595] req@ffff8800a73cf100 x1818247332827520/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017595.543200:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332827520/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017595.543208:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017595.543211:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017595.543213:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332827520:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017595.896488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017596.898492:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017597.900492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017598.902509:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017599.904509:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017600.550340:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017600.550350:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017600.550354:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017600.550365:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1863 00000100:00080000:3.0:1734017600.550375:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017600.550383:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017600.550419:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332827648/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017600.550451:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017600.550456:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017600.550458:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017600.550462:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1863 00000100:00080000:3.0:1734017600.550466:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017600.550470:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017600.550480:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017600.550490:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332827776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017600.550500:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017600.550504:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017600.550506:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017600.550510:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1863 00000100:00080000:3.0:1734017600.550513:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017600.550517:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017600.550522:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017600.550530:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332827904/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017600.550538:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017600.550541:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017600.550542:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017600.550546:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1863 00000100:00080000:3.0:1734017600.550548:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017600.550552:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017600.550556:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017600.550562:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332827648:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017600.550564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332828032/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017600.550572:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017600.550576:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017600.550577:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017600.550581:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1863 00000100:00080000:3.0:1734017600.550583:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017600.550587:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017600.550591:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017600.550598:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332828160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017600.550613:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017600.550616:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017600.550618:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017600.550622:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1863 00000100:00080000:3.0:1734017600.550624:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017600.550628:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017600.550632:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017600.550639:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332828288/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017600.550647:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017600.550650:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017600.550651:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017600.550655:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332827776:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017600.550655:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1863 00000100:00080000:3.0:1734017600.550657:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017600.550660:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017600.550665:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017600.550674:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332828416/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017600.550676:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332827904:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017600.550699:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332828032:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017600.550714:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332828160:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017600.550729:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332828288:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017600.550744:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332828416:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017600.906472:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017601.908479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017602.910479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734017603.556612:0:1958:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734017603.556666:0:1958:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734017603.556685:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017603.556713:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017603.556721:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017603.556728:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017603.556734:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:2.0:1734017603.556736:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017600/real 1734017603] req@ffff8800a73cc380 x1818247332827648/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000800:00000100:0.0:1734017603.556772:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017603.556778:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:2.0:1734017603.556796:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332827648/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017603.556810:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017603.556816:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017603.556824:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332827648:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017603.556835:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017600/real 1734017603] req@ffff8800a73cca80 x1818247332827776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017603.556855:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332827776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017603.556868:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017603.556872:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017603.556878:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332827776:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017603.556887:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017600/real 1734017603] req@ffff8800a73cf480 x1818247332827904/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017603.556906:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332827904/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017603.556919:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017603.556924:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017603.556930:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332827904:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017603.556961:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017600/real 1734017603] req@ffff8800a73cd880 x1818247332828032/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017603.556981:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332828032/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017603.557002:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017603.557006:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017603.557011:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332828032:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017603.557021:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017600/real 1734017603] req@ffff8800a73cea00 x1818247332828160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017603.557038:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332828160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017603.557050:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017603.557054:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017603.557060:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332828160:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017603.557069:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017600/real 1734017603] req@ffff8800a73ced80 x1818247332828288/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017603.557087:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332828288/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017603.557099:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017603.557103:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017603.557109:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332828288:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017603.557119:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017600/real 1734017603] req@ffff8800a73cce00 x1818247332828416/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017603.557138:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332828416/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017603.557161:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017603.557165:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017603.557170:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332828416:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017603.912515:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017604.914504:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017605.558356:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017605.558365:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017605.558370:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017605.558381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1868 00000100:00080000:3.0:1734017605.558391:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017605.558399:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017605.558435:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332828544/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017605.558469:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017605.558473:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017605.558475:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017605.558479:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1868 00000100:00080000:3.0:1734017605.558482:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017605.558486:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017605.558497:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017605.558508:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332828672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017605.558518:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017605.558521:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017605.558523:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017605.558527:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1868 00000100:00080000:3.0:1734017605.558530:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017605.558533:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017605.558538:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017605.558546:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332828800/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017605.558554:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017605.558558:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017605.558560:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017605.558564:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1868 00000100:00080000:3.0:1734017605.558566:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017605.558570:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017605.558574:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017605.558575:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332828544:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017605.558582:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332828928/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017605.558591:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017605.558594:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017605.558595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017605.558599:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1868 00000100:00080000:3.0:1734017605.558601:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017605.558605:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017605.558609:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017605.558616:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332829056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017605.558624:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017605.558627:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017605.558629:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017605.558633:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1868 00000100:00080000:3.0:1734017605.558636:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734017605.558639:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017605.558639:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017605.558644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017605.558651:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332829184/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734017605.558660:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017605/real 1734017605] req@ffff8800a73cd500 x1818247332828544/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017605.558660:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017605.558663:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017605.558664:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017605.558668:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1868 00000100:00080000:3.0:1734017605.558670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017605.558673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017605.558678:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017605.558680:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332828544/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017605.558686:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332829312/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017605.558691:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017605.558694:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017605.558699:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332828544:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017605.558738:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332828672:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017605.558759:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017605.558768:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017605/real 1734017605] req@ffff8800a73cce00 x1818247332828672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017605.558779:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332828672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017605.558787:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017605.558790:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017605.558794:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332828672:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017605.558801:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332828800:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017605.558815:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017605.558821:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017605/real 1734017605] req@ffff8800a73ced80 x1818247332828800/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017605.558831:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332828800/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017605.558839:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017605.558841:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017605.558844:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332828800:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017605.558868:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332828928:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017605.558881:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017605.558887:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017605/real 1734017605] req@ffff8800a73cea00 x1818247332828928/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017605.558897:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332828928/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017605.558905:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017605.558907:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017605.558910:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332828928:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017605.558915:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332829056:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017605.558926:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017605.558933:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017605/real 1734017605] req@ffff8800a73cd880 x1818247332829056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017605.558943:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332829056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017605.558951:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017605.558953:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017605.558956:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332829056:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017605.558961:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332829184:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017605.558972:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017605.558978:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017605/real 1734017605] req@ffff8800a73cf480 x1818247332829184/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017605.558988:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332829184/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017605.558995:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017605.558997:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017605.559000:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332829184:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017605.559005:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332829312:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017605.559016:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017605.559022:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017605/real 1734017605] req@ffff8800a73cca80 x1818247332829312/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017605.559031:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332829312/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017605.559046:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017605.559049:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017605.559052:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332829312:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017605.916491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017606.918511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017607.920508:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017608.922463:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017609.924473:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017610.566338:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017610.566348:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017610.566352:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017610.566363:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1873 00000100:00080000:3.0:1734017610.566371:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017610.566380:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017610.566416:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332829440/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017610.566450:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017610.566457:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017610.566459:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017610.566464:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1873 00000100:00080000:3.0:1734017610.566467:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017610.566471:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017610.566481:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017610.566491:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332829568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017610.566501:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017610.566504:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017610.566506:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017610.566510:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1873 00000100:00080000:3.0:1734017610.566512:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017610.566516:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017610.566521:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017610.566529:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332829696/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017610.566537:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017610.566540:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017610.566542:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017610.566545:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1873 00000100:00080000:3.0:1734017610.566547:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017610.566551:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017610.566556:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017610.566558:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332829440:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017610.566564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332829824/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017610.566572:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017610.566576:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017610.566577:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017610.566580:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1873 00000100:00080000:3.0:1734017610.566583:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017610.566586:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017610.566590:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017610.566597:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332829952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017610.566605:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017610.566608:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017610.566610:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017610.566613:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1873 00000100:00080000:3.0:1734017610.566616:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017610.566650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017610.566655:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017610.566664:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332830080/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734017610.566665:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017610.566673:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017610.566676:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017610.566678:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017610.566682:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1873 00000100:00080000:3.0:1734017610.566685:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734017610.566689:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017610/real 1734017610] req@ffff8800a73cc380 x1818247332829440/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017610.566689:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017610.566693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017610.566702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332830208/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734017610.566708:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332829440/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017610.566719:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017610.566722:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017610.566727:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332829440:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017610.566734:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332829568:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017610.566754:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017610.566761:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017610/real 1734017610] req@ffff8800a73cf100 x1818247332829568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017610.566772:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332829568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017610.566780:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017610.566782:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017610.566787:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332829568:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017610.566793:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332829696:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017610.566806:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017610.566812:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017610/real 1734017610] req@ffff8800a73cdf80 x1818247332829696/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017610.566822:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332829696/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017610.566829:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017610.566832:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017610.566835:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332829696:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017610.566860:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332829824:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017610.566873:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017610.566879:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017610/real 1734017610] req@ffff8800a73cf800 x1818247332829824/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017610.566889:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332829824/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017610.566897:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017610.566899:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017610.566902:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332829824:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017610.566907:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332829952:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017610.566919:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017610.566926:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017610/real 1734017610] req@ffff8800a73cfb80 x1818247332829952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017610.566935:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332829952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017610.566943:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017610.566945:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017610.566948:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332829952:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017610.566953:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332830080:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017610.566966:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017610.566971:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017610/real 1734017610] req@ffff8800a73cdc00 x1818247332830080/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017610.566981:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332830080/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017610.566988:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017610.566990:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017610.566993:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332830080:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017610.566998:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332830208:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017610.567010:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017610.567016:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017610/real 1734017610] req@ffff8800a73ce680 x1818247332830208/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017610.567026:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332830208/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017610.567040:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017610.567043:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017610.567046:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332830208:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017610.926488:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017611.928505:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017612.930488:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017613.932469:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017614.934473:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017615.574352:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017615.574362:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017615.574366:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017615.574376:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1878 00000100:00080000:3.0:1734017615.574386:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017615.574394:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017615.574431:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332830336/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017615.574466:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017615.574471:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017615.574473:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017615.574477:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1878 00000100:00080000:3.0:1734017615.574481:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017615.574485:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017615.574494:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017615.574507:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332830464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017615.574518:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017615.574521:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017615.574523:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017615.574526:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1878 00000100:00080000:3.0:1734017615.574529:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017615.574532:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017615.574567:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017615.574578:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332830592/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017615.574587:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017615.574590:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017615.574591:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017615.574595:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1878 00000100:00080000:3.0:1734017615.574598:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017615.574601:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017615.574605:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017615.574614:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332830720/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734017615.574615:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332830336:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017615.574623:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017615.574626:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017615.574628:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017615.574632:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1878 00000100:00080000:3.0:1734017615.574635:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017615.574638:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017615.574643:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017615.574650:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332830848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017615.574659:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017615.574662:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017615.574663:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017615.574667:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1878 00000100:00080000:3.0:1734017615.574670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017615.574673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017615.574677:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734017615.574680:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017615.574685:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332830976/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017615.574693:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017615.574696:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017615.574697:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017615.574701:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1878 00000100:00100000:2.0:1734017615.574703:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017615/real 1734017615] req@ffff8800a73cc700 x1818247332830336/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017615.574704:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017615.574707:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017615.574712:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017615.574719:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332831104/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734017615.574722:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332830336/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017615.574734:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017615.574737:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017615.574742:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332830336:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017615.574749:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332830464:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017615.574767:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017615.574773:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017615/real 1734017615] req@ffff8800a73ce680 x1818247332830464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017615.574784:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332830464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017615.574792:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017615.574795:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017615.574799:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332830464:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017615.574805:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332830592:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017615.574819:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017615.574825:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017615/real 1734017615] req@ffff8800a73cdc00 x1818247332830592/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017615.574835:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332830592/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017615.574843:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017615.574845:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017615.574849:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332830592:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017615.574872:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332830720:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017615.574884:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017615.574890:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017615/real 1734017615] req@ffff8800a73cfb80 x1818247332830720/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017615.574901:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332830720/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017615.574909:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017615.574911:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017615.574914:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332830720:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017615.574919:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332830848:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017615.574934:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017615.574940:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017615/real 1734017615] req@ffff8800a73cf800 x1818247332830848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017615.574951:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332830848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017615.574958:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017615.574960:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017615.574963:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332830848:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017615.574968:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332830976:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017615.574980:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017615.574986:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017615/real 1734017615] req@ffff8800a73cdf80 x1818247332830976/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017615.574996:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332830976/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017615.575003:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017615.575006:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017615.575009:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332830976:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017615.575014:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332831104:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017615.575026:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017615.575032:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017615/real 1734017615] req@ffff8800a73cf100 x1818247332831104/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017615.575049:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332831104/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017615.575056:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017615.575059:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017615.575061:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332831104:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017615.936484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017616.938507:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017617.940443:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017618.942537:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017619.944450:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017620.582357:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017620.582367:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017620.582371:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017620.582381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1883 00000100:00080000:3.0:1734017620.582390:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017620.582398:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017620.582434:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332831232/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017620.582500:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017620.582504:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017620.582506:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017620.582511:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1883 00000100:00080000:3.0:1734017620.582514:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017620.582518:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017620.582528:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017620.582540:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332831360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017620.582550:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017620.582553:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017620.582555:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017620.582559:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1883 00000100:00080000:3.0:1734017620.582562:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017620.582566:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017620.582570:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017620.582579:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332831488/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017620.582587:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017620.582590:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017620.582592:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017620.582595:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1883 00000100:00080000:3.0:1734017620.582597:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017620.582600:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017620.582605:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017620.582612:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332831616/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734017620.582620:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332831232:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017620.582621:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017620.582624:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017620.582626:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017620.582629:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1883 00000100:00080000:3.0:1734017620.582632:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017620.582635:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017620.582639:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017620.582647:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332831744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017620.582661:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017620.582665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017620.582666:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017620.582670:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1883 00000100:00080000:3.0:1734017620.582682:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017620.582686:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:2.0:1734017620.582687:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017620.582690:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017620.582698:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332831872/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017620.582706:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734017620.582710:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017620/real 1734017620] req@ffff8800a73cc380 x1818247332831232/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017620.582710:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017620.582711:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017620.582715:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1883 00000100:00080000:3.0:1734017620.582718:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017620.582721:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017620.582725:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017620.582730:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332831232/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017620.582734:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332832000/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017620.582741:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017620.582744:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017620.582749:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332831232:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017620.582756:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332831360:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017620.582773:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017620.582780:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017620/real 1734017620] req@ffff8800a73cca80 x1818247332831360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017620.582791:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332831360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017620.582800:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017620.582803:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017620.582806:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332831360:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017620.582812:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332831488:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017620.582826:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017620.582831:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017620/real 1734017620] req@ffff8800a73cf480 x1818247332831488/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017620.582841:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332831488/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017620.582849:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017620.582851:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017620.582854:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332831488:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017620.582879:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332831616:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017620.582892:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017620.582898:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017620/real 1734017620] req@ffff8800a73cd880 x1818247332831616/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017620.582909:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332831616/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017620.582916:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017620.582918:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017620.582921:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332831616:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017620.582928:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332831744:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017620.582940:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017620.582946:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017620/real 1734017620] req@ffff8800a73cea00 x1818247332831744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017620.582957:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332831744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017620.582964:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017620.582967:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017620.582970:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332831744:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017620.582975:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332831872:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017620.582986:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017620.582992:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017620/real 1734017620] req@ffff8800a73ced80 x1818247332831872/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017620.583002:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332831872/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017620.583009:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017620.583011:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017620.583014:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332831872:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017620.583019:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332832000:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017620.583032:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017620.583038:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017620/real 1734017620] req@ffff8800a73cce00 x1818247332832000/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017620.583049:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332832000/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017620.583056:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017620.583058:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017620.583061:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332832000:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017620.946505:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017621.948511:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017622.950496:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017623.952483:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017624.954528:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017625.590353:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017625.590363:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017625.590409:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017625.590419:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1888 00000100:00080000:3.0:1734017625.590429:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017625.590437:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017625.590474:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332832128/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017625.590505:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017625.590509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017625.590512:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017625.590516:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1888 00000100:00080000:3.0:1734017625.590519:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017625.590523:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017625.590533:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017625.590545:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332832256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017625.590555:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017625.590559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017625.590560:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017625.590564:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1888 00000100:00080000:3.0:1734017625.590566:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017625.590570:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017625.590575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017625.590583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332832384/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017625.590591:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017625.590595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017625.590597:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017625.590600:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1888 00000100:00080000:3.0:1734017625.590602:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017625.590606:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017625.590610:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017625.590616:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332832128:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017625.590618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332832512/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017625.590627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017625.590630:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017625.590631:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017625.590635:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1888 00000100:00080000:3.0:1734017625.590637:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017625.590641:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017625.590645:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017625.590652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332832640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017625.590660:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017625.590663:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017625.590665:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017625.590669:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1888 00000100:00080000:3.0:1734017625.590673:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017625.590676:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017625.590681:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734017625.590683:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017625.590688:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332832768/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017625.590697:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017625.590700:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017625.590701:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017625.590705:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1888 00000100:00100000:2.0:1734017625.590706:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017625/real 1734017625] req@ffff8800a73cd500 x1818247332832128/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017625.590708:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017625.590711:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017625.590715:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017625.590723:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332832896/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734017625.590726:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332832128/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017625.590738:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017625.590741:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017625.590746:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332832128:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017625.590753:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332832256:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017625.590771:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017625.590778:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017625/real 1734017625] req@ffff8800a73cce00 x1818247332832256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017625.590789:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332832256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017625.590798:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017625.590801:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017625.590804:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332832256:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017625.590810:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332832384:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017625.590823:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017625.590829:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017625/real 1734017625] req@ffff8800a73ced80 x1818247332832384/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017625.590839:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332832384/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017625.590847:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017625.590849:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017625.590852:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332832384:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017625.590877:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332832512:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017625.590889:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017625.590896:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017625/real 1734017625] req@ffff8800a73cea00 x1818247332832512/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017625.590906:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332832512/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017625.590914:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017625.590916:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017625.590919:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332832512:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017625.590925:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332832640:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017625.590938:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017625.590944:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017625/real 1734017625] req@ffff8800a73cd880 x1818247332832640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017625.590954:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332832640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017625.590962:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017625.590964:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017625.590967:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332832640:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017625.590973:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332832768:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017625.590984:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017625.590990:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017625/real 1734017625] req@ffff8800a73cf480 x1818247332832768/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017625.590999:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332832768/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017625.591006:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017625.591009:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017625.591011:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332832768:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017625.591017:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332832896:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017625.591029:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017625.591035:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017625/real 1734017625] req@ffff8800a73cca80 x1818247332832896/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017625.591045:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332832896/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017625.591053:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017625.591055:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017625.591058:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332832896:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017625.956472:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017626.958546:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017627.960559:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017628.962546:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017629.964514:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017630.598421:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017630.598430:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017630.598435:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017630.598446:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1893 00000100:00080000:3.0:1734017630.598457:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017630.598465:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017630.598500:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332833024/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017630.598534:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017630.598541:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017630.598544:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017630.598548:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1893 00000100:00080000:3.0:1734017630.598551:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017630.598555:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017630.598565:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017630.598575:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332833152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017630.598585:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017630.598588:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017630.598590:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017630.598594:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1893 00000100:00080000:3.0:1734017630.598596:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017630.598600:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017630.598605:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017630.598612:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332833280/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017630.598621:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017630.598624:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017630.598625:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017630.598629:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1893 00000100:00080000:3.0:1734017630.598631:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017630.598635:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017630.598640:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017630.598648:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332833408/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734017630.598653:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332833024:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017630.598657:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017630.598660:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017630.598662:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017630.598665:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1893 00000100:00080000:3.0:1734017630.598668:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017630.598671:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017630.598675:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017630.598682:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332833536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017630.598690:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017630.598693:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017630.598695:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017630.598699:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1893 00000100:00080000:3.0:1734017630.598711:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017630.598715:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:2.0:1734017630.598719:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017630.598719:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017630.598727:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332833664/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017630.598735:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017630.598738:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1734017630.598741:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017630/real 1734017630] req@ffff8800a73cc380 x1818247332833024/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017630.598741:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017630.598745:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1893 00000100:00080000:3.0:1734017630.598748:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017630.598751:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017630.598755:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017630.598761:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332833024/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017630.598765:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332833792/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017630.598772:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017630.598776:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017630.598781:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332833024:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017630.598788:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332833152:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017630.598805:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017630.598811:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017630/real 1734017630] req@ffff8800a73cf100 x1818247332833152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017630.598822:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332833152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017630.598832:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017630.598834:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017630.598837:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332833152:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017630.598843:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332833280:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017630.598857:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017630.598862:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017630/real 1734017630] req@ffff8800a73cdf80 x1818247332833280/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017630.598872:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332833280/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017630.598880:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017630.598883:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017630.598886:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332833280:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017630.598911:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332833408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017630.598924:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017630.598930:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017630/real 1734017630] req@ffff8800a73cf800 x1818247332833408/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017630.598940:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332833408/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017630.598948:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017630.598950:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017630.598953:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332833408:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017630.598959:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332833536:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017630.598971:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017630.598977:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017630/real 1734017630] req@ffff8800a73cfb80 x1818247332833536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017630.598988:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332833536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017630.598996:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017630.598998:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017630.599001:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332833536:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017630.599007:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332833664:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017630.599018:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017630.599024:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017630/real 1734017630] req@ffff8800a73cdc00 x1818247332833664/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017630.599033:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332833664/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017630.599040:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017630.599043:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017630.599046:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332833664:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017630.599051:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332833792:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017630.599063:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017630.599069:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017630/real 1734017630] req@ffff8800a73ce680 x1818247332833792/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017630.599079:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332833792/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017630.599087:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017630.599089:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017630.599092:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332833792:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017630.966540:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017631.968515:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017632.970470:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017633.972471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017634.974481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017635.606359:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017635.606368:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017635.606372:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017635.606382:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1898 00000100:00080000:3.0:1734017635.606392:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017635.606400:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017635.606436:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332833920/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017635.606468:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017635.606473:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017635.606475:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017635.606479:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1898 00000100:00080000:3.0:1734017635.606482:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017635.606486:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017635.606496:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017635.606508:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332834048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017635.606519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017635.606523:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017635.606524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017635.606527:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1898 00000100:00080000:3.0:1734017635.606530:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017635.606533:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017635.606538:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017635.606546:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332834176/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017635.606554:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017635.606557:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017635.606559:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017635.606562:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1898 00000100:00080000:3.0:1734017635.606565:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017635.606568:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017635.606572:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017635.606579:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332834304/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017635.606588:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017635.606591:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1734017635.606592:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332833920:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017635.606593:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017635.606596:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1898 00000100:00080000:3.0:1734017635.606600:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017635.606603:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017635.606608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017635.606616:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332834432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017635.606624:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017635.606626:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017635.606628:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017635.606632:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1898 00000100:00080000:3.0:1734017635.606634:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017635.606637:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017635.606642:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017635.606649:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332834560/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017635.606657:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017635.606660:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017635.606661:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017635.606664:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1898 00000100:00080000:3.0:1734017635.606667:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017635.606670:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:2.0:1734017635.606671:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017635.606674:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017635.606682:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332834688/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734017635.606695:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017635/real 1734017635] req@ffff8800a73cc700 x1818247332833920/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017635.606715:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332833920/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017635.606726:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017635.606729:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017635.606734:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332833920:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017635.606742:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332834048:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017635.606762:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017635.606769:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017635/real 1734017635] req@ffff8800a73ce680 x1818247332834048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017635.606780:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332834048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017635.606788:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017635.606791:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017635.606795:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332834048:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017635.606801:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332834176:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017635.606814:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017635.606820:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017635/real 1734017635] req@ffff8800a73cdc00 x1818247332834176/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017635.606830:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332834176/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017635.606838:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017635.606840:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017635.606843:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332834176:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017635.606867:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332834304:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017635.606880:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017635.606886:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017635/real 1734017635] req@ffff8800a73cfb80 x1818247332834304/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017635.606896:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332834304/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017635.606904:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017635.606906:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017635.606909:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332834304:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017635.606915:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332834432:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017635.606927:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017635.606933:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017635/real 1734017635] req@ffff8800a73cf800 x1818247332834432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017635.606944:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332834432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017635.606951:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017635.606953:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017635.606956:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332834432:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017635.606962:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332834560:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017635.606973:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017635.606978:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017635/real 1734017635] req@ffff8800a73cdf80 x1818247332834560/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017635.606988:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332834560/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017635.606995:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017635.606997:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017635.607000:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332834560:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017635.607005:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332834688:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017635.607017:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017635.607023:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017635/real 1734017635] req@ffff8800a73cf100 x1818247332834688/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017635.607033:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332834688/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017635.607041:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017635.607043:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017635.607046:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332834688:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017635.976475:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017636.978479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017637.980484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017638.982499:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017639.984480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017640.614357:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017640.614366:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017640.614371:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017640.614381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1903 00000100:00080000:3.0:1734017640.614390:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017640.614398:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017640.614433:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332834816/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017640.614467:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017640.614471:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017640.614473:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017640.614478:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1903 00000100:00080000:3.0:1734017640.614481:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017640.614485:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017640.614495:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017640.614506:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332834944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017640.614516:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017640.614519:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017640.614521:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017640.614524:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1903 00000100:00080000:3.0:1734017640.614528:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017640.614531:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017640.614536:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017640.614544:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332835072/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017640.614552:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017640.614555:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017640.614557:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017640.614560:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1903 00000100:00080000:3.0:1734017640.614563:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017640.614567:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:2.0:1734017640.614570:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332834816:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017640.614571:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017640.614578:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332835200/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017640.614587:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017640.614590:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017640.614592:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017640.614595:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1903 00000100:00080000:3.0:1734017640.614597:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017640.614601:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017640.614605:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017640.614613:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332835328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017640.614628:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017640.614631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017640.614632:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1734017640.614635:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017640.614636:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1903 00000100:00080000:3.0:1734017640.614639:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017640.614642:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017640.614646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017640.614654:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332835456/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734017640.614658:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017640/real 1734017640] req@ffff8800a73cc380 x1818247332834816/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017640.614662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017640.614665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017640.614666:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017640.614670:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1903 00000100:00080000:3.0:1734017640.614672:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017640.614675:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:2.0:1734017640.614677:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332834816/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017640.614679:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1734017640.614688:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1734017640.614689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332835584/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017640.614691:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017640.614696:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332834816:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017640.614703:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332834944:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017640.614721:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017640.614727:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017640/real 1734017640] req@ffff8800a73cca80 x1818247332834944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017640.614739:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332834944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017640.614747:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017640.614750:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017640.614754:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332834944:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017640.614760:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332835072:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017640.614773:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017640.614779:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017640/real 1734017640] req@ffff8800a73cf480 x1818247332835072/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017640.614789:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332835072/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017640.614796:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017640.614798:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017640.614802:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332835072:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017640.614827:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332835200:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017640.614839:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017640.614845:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017640/real 1734017640] req@ffff8800a73cd880 x1818247332835200/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017640.614855:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332835200/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017640.614863:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017640.614865:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017640.614868:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332835200:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017640.614874:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332835328:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017640.614886:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017640.614892:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017640/real 1734017640] req@ffff8800a73cea00 x1818247332835328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017640.614902:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332835328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017640.614910:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017640.614912:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017640.614915:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332835328:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017640.614920:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332835456:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017640.614931:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017640.614937:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017640/real 1734017640] req@ffff8800a73ced80 x1818247332835456/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017640.614947:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332835456/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017640.614954:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017640.614957:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017640.614960:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332835456:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017640.614965:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332835584:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017640.614977:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017640.614983:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017640/real 1734017640] req@ffff8800a73cce00 x1818247332835584/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017640.614993:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332835584/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017640.615001:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017640.615003:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017640.615006:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332835584:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017640.986449:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017641.988486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017642.990487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017643.992487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017644.994484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017645.622388:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017645.622397:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017645.622402:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017645.622413:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1908 00000100:00080000:3.0:1734017645.622422:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017645.622430:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017645.622466:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332835712/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017645.622499:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017645.622503:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017645.622505:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017645.622510:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1908 00000100:00080000:3.0:1734017645.622513:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017645.622517:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017645.622526:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017645.622539:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332835840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017645.622549:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017645.622552:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017645.622554:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017645.622558:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1908 00000100:00080000:3.0:1734017645.622560:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017645.622564:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017645.622569:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017645.622577:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332835968/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017645.622585:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017645.622589:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017645.622591:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017645.622595:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1908 00000100:00080000:3.0:1734017645.622597:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017645.622601:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:2.0:1734017645.622605:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332835712:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017645.622605:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017645.622613:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332836096/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017645.622621:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017645.622625:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017645.622626:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017645.622630:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1908 00000100:00080000:3.0:1734017645.622632:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017645.622635:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017645.622640:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017645.622647:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332836224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017645.622655:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017645.622658:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017645.622660:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017645.622664:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1908 00000100:00080000:3.0:1734017645.622666:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017645.622670:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:2.0:1734017645.622671:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017645.622675:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017645.622682:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332836352/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017645.622690:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734017645.622694:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017645/real 1734017645] req@ffff8800a73cd500 x1818247332835712/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017645.622694:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017645.622695:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017645.622699:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1908 00000100:00080000:3.0:1734017645.622701:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017645.622704:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017645.622708:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017645.622714:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332835712/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017645.622717:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332836480/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017645.622725:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017645.622729:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017645.622734:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332835712:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017645.622741:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332835840:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017645.622758:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017645.622764:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017645/real 1734017645] req@ffff8800a73cce00 x1818247332835840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017645.622775:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332835840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017645.622784:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017645.622787:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017645.622790:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332835840:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017645.622796:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332835968:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017645.622809:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017645.622815:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017645/real 1734017645] req@ffff8800a73ced80 x1818247332835968/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017645.622825:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332835968/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017645.622832:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017645.622834:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017645.622837:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332835968:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017645.622862:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332836096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017645.622875:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017645.622881:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017645/real 1734017645] req@ffff8800a73cea00 x1818247332836096/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017645.622891:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332836096/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017645.622898:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017645.622901:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017645.622904:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332836096:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017645.622911:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332836224:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017645.622923:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017645.622929:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017645/real 1734017645] req@ffff8800a73cd880 x1818247332836224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017645.622939:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332836224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017645.622947:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017645.622949:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017645.622952:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332836224:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017645.622958:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332836352:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017645.622969:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017645.622975:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017645/real 1734017645] req@ffff8800a73cf480 x1818247332836352/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017645.622984:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332836352/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017645.622991:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017645.622994:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017645.622997:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332836352:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017645.623002:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332836480:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017645.623014:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017645.623020:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017645/real 1734017645] req@ffff8800a73cca80 x1818247332836480/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017645.623076:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332836480/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017645.623084:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017645.623087:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017645.623090:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332836480:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017645.996470:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017646.998495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017648.000488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017649.002511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017650.004474:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017650.630360:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017650.630369:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017650.630373:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017650.630383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1913 00000100:00080000:3.0:1734017650.630392:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017650.630400:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017650.630435:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332836608/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017650.630469:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017650.630476:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017650.630479:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017650.630483:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1913 00000100:00080000:3.0:1734017650.630486:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017650.630490:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017650.630500:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017650.630511:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332836736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017650.630521:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017650.630524:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017650.630525:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017650.630529:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1913 00000100:00080000:3.0:1734017650.630532:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017650.630536:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017650.630541:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017650.630549:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332836864/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017650.630559:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017650.630562:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017650.630564:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017650.630567:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1913 00000100:00080000:3.0:1734017650.630570:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734017650.630573:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332836608:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017650.630573:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017650.630578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017650.630585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332836992/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017650.630593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017650.630597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017650.630598:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017650.630601:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1913 00000100:00080000:3.0:1734017650.630604:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017650.630607:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017650.630611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017650.630618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332837120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017650.630626:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017650.630629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017650.630631:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017650.630634:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1913 00000100:00080000:3.0:1734017650.630637:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734017650.630639:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017650.630641:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017650.630645:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017650.630652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332837248/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017650.630660:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734017650.630661:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017650/real 1734017650] req@ffff8800a73cc380 x1818247332836608/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017650.630663:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017650.630665:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017650.630669:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1913 00000100:00080000:3.0:1734017650.630672:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017650.630675:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017650.630679:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017650.630680:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332836608/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017650.630689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332837376/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017650.630692:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017650.630695:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017650.630700:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332836608:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017650.630707:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332836736:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017650.630724:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017650.630730:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017650/real 1734017650] req@ffff8800a73cf100 x1818247332836736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017650.630741:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332836736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017650.630750:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017650.630752:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017650.630756:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332836736:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017650.630779:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332836864:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017650.630793:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017650.630799:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017650/real 1734017650] req@ffff8800a73cdf80 x1818247332836864/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017650.630809:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332836864/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017650.630817:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017650.630819:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017650.630822:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332836864:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017650.630828:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332836992:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017650.630839:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017650.630844:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017650/real 1734017650] req@ffff8800a73cf800 x1818247332836992/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017650.630854:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332836992/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017650.630861:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017650.630863:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017650.630867:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332836992:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017650.630873:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332837120:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017650.630885:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017650.630891:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017650/real 1734017650] req@ffff8800a73cfb80 x1818247332837120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017650.630901:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332837120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017650.630909:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017650.630912:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017650.630914:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332837120:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017650.630920:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332837248:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017650.630932:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017650.630973:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017650/real 1734017650] req@ffff8800a73cdc00 x1818247332837248/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017650.630987:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332837248/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017650.630994:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017650.630996:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017650.630999:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332837248:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017650.631005:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332837376:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017650.631020:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017650.631027:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017650/real 1734017650] req@ffff8800a73ce680 x1818247332837376/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017650.631038:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332837376/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017650.631046:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017650.631048:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017650.631051:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332837376:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017651.006484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017652.008458:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017653.010490:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017654.012480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017655.014447:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017655.638324:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017655.638335:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017655.638340:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017655.638351:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1918 00000100:00080000:3.0:1734017655.638360:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017655.638367:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017655.638402:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332837504/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017655.638437:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017655.638441:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017655.638443:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017655.638448:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1918 00000100:00080000:3.0:1734017655.638451:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017655.638455:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017655.638464:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017655.638476:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332837632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017655.638487:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017655.638490:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017655.638492:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017655.638495:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1918 00000100:00080000:3.0:1734017655.638498:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017655.638501:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017655.638506:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017655.638514:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332837760/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017655.638523:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017655.638526:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017655.638527:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017655.638531:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1918 00000100:00080000:3.0:1734017655.638533:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017655.638536:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017655.638541:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017655.638549:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332837504:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017655.638549:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332837888/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017655.638558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017655.638561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017655.638563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017655.638566:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1918 00000100:00080000:3.0:1734017655.638569:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017655.638573:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017655.638577:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017655.638585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332838016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017655.638593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017655.638596:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017655.638597:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017655.638601:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1918 00000100:00080000:3.0:1734017655.638604:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017655.638607:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017655.638611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734017655.638613:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017655.638618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332838144/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017655.638626:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017655.638629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017655.638631:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017655.638634:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1918 00000100:00100000:2.0:1734017655.638636:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017655/real 1734017655] req@ffff8800a73cc700 x1818247332837504/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017655.638637:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017655.638640:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017655.638644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017655.638652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332838272/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734017655.638656:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332837504/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017655.638667:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017655.638670:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017655.638674:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332837504:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017655.638682:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332837632:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017655.638698:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017655.638705:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017655/real 1734017655] req@ffff8800a73ce680 x1818247332837632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017655.638716:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332837632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017655.638725:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017655.638727:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017655.638731:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332837632:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017655.638736:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332837760:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017655.638750:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017655.638755:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017655/real 1734017655] req@ffff8800a73cdc00 x1818247332837760/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017655.638765:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332837760/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017655.638773:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017655.638775:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017655.638778:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332837760:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017655.638803:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332837888:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017655.638815:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017655.638822:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017655/real 1734017655] req@ffff8800a73cfb80 x1818247332837888/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017655.638832:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332837888/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017655.638839:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017655.638841:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017655.638844:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332837888:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017655.638851:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332838016:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017655.638913:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017655.638921:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017655/real 1734017655] req@ffff8800a73cf800 x1818247332838016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017655.638931:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332838016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017655.638940:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017655.638942:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017655.638945:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332838016:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017655.638951:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332838144:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017655.638963:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017655.638968:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017655/real 1734017655] req@ffff8800a73cdf80 x1818247332838144/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017655.638978:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332838144/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017655.638986:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017655.638988:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017655.638991:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332838144:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017655.638996:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332838272:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017655.639008:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017655.639013:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017655/real 1734017655] req@ffff8800a73cf100 x1818247332838272/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017655.639024:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332838272/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017655.639032:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017655.639034:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017655.639037:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332838272:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017656.016487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017657.018479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017658.020449:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017659.022390:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017660.024376:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017660.646361:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017660.646370:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017660.646374:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017660.646385:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1923 00000100:00080000:3.0:1734017660.646395:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017660.646403:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017660.646437:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332838400/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017660.646471:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017660.646475:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017660.646477:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017660.646481:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1923 00000100:00080000:3.0:1734017660.646484:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017660.646488:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017660.646497:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017660.646508:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332838528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017660.646518:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017660.646521:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017660.646523:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017660.646527:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1923 00000100:00080000:3.0:1734017660.646530:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017660.646534:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017660.646539:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017660.646547:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332838656/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017660.646556:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017660.646559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017660.646560:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017660.646563:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1923 00000100:00080000:3.0:1734017660.646566:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017660.646569:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017660.646573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017660.646576:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332838400:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017660.646581:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332838784/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017660.646590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017660.646593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017660.646595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017660.646598:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1923 00000100:00080000:3.0:1734017660.646601:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017660.646604:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017660.646608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017660.646616:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332838912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017660.646630:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017660.646633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017660.646635:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017660.646638:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1923 00000100:00080000:3.0:1734017660.646641:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734017660.646643:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017660.646645:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017660.646649:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017660.646656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332839040/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017660.646664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734017660.646666:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017660/real 1734017660] req@ffff8800a73cc380 x1818247332838400/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017660.646667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017660.646669:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017660.646672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1923 00000100:00080000:3.0:1734017660.646675:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017660.646678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017660.646682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017660.646686:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332838400/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017660.646691:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332839168/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017660.646697:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017660.646700:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017660.646705:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332838400:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017660.646712:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332838528:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017660.646729:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017660.646735:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017660/real 1734017660] req@ffff8800a73cca80 x1818247332838528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017660.646746:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332838528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017660.646755:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017660.646758:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017660.646761:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332838528:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017660.646818:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332838656:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017660.646837:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017660.646845:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017660/real 1734017660] req@ffff8800a73cf480 x1818247332838656/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017660.646855:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332838656/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017660.646863:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017660.646866:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017660.646869:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332838656:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017660.646895:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332838784:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017660.646907:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017660.646913:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017660/real 1734017660] req@ffff8800a73cd880 x1818247332838784/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017660.646923:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332838784/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017660.646931:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017660.646934:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017660.646937:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332838784:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017660.646943:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332838912:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017660.646955:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017660.646961:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017660/real 1734017660] req@ffff8800a73cea00 x1818247332838912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017660.646971:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332838912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017660.646979:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017660.646981:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017660.646984:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332838912:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017660.646990:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332839040:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017660.647001:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017660.647007:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017660/real 1734017660] req@ffff8800a73ced80 x1818247332839040/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017660.647017:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332839040/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017660.647024:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017660.647026:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017660.647029:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332839040:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017660.647034:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332839168:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017660.647046:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017660.647052:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017660/real 1734017660] req@ffff8800a73cce00 x1818247332839168/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017660.647063:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332839168/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017660.647070:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017660.647072:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017660.647075:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332839168:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017661.026415:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017662.028370:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017663.030395:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017664.032464:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017665.034482:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017665.654353:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017665.654362:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017665.654366:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017665.654377:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1928 00000100:00080000:3.0:1734017665.654387:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017665.654395:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017665.654433:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332839296/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017665.654466:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017665.654470:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017665.654472:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017665.654477:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1928 00000100:00080000:3.0:1734017665.654480:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017665.654484:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017665.654494:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017665.654506:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332839424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017665.654516:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017665.654519:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017665.654521:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017665.654525:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1928 00000100:00080000:3.0:1734017665.654527:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017665.654531:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017665.654535:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017665.654544:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332839552/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017665.654552:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017665.654556:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017665.654557:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017665.654561:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1928 00000100:00080000:3.0:1734017665.654563:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017665.654567:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017665.654571:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017665.654576:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332839296:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017665.654579:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332839680/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017665.654588:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017665.654591:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017665.654592:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017665.654596:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1928 00000100:00080000:3.0:1734017665.654598:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017665.654601:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017665.654606:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017665.654613:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332839808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017665.654621:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017665.654624:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017665.654626:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017665.654629:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1928 00000100:00080000:3.0:1734017665.654632:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017665.654636:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017665.654641:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017665.654648:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332839936/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017665.654656:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017665.654659:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017665.654661:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017665.654665:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1928 00000100:00100000:2.0:1734017665.654666:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332839424:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017665.654667:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017665.654670:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017665.654674:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017665.654708:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332840064/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734017665.654717:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332839552:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017665.654743:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332839680:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017665.654757:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332839808:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017665.654771:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332839936:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017665.654784:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332840064:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017666.036482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017667.038468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017668.040448:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734017668.660468:0:1956:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734017668.660514:0:1956:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734017668.660533:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017668.660572:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017668.660584:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017668.660596:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017668.660606:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017668.660616:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017668.660625:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:3.0:1734017668.660671:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017665/real 1734017668] req@ffff8800a73cea00 x1818247332839680/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017668.660705:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332839680/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017668.660723:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017668.660729:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017668.660736:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332839680:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734017668.660747:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017665/real 1734017668] req@ffff8800a73cd880 x1818247332839808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017668.660778:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332839808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017668.660791:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017668.660795:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017668.660801:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332839808:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734017668.660810:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017665/real 1734017668] req@ffff8800a73cf480 x1818247332839936/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017668.660828:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332839936/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017668.660841:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017668.660845:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017668.660851:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332839936:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734017668.660861:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017665/real 1734017668] req@ffff8800a73cca80 x1818247332840064/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017668.660879:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332840064/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017668.660891:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017668.660906:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017668.660913:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332840064:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734017668.660922:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017665/real 1734017668] req@ffff8800a73cd500 x1818247332839296/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017668.660953:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332839296/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017668.660965:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017668.660969:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017668.660974:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332839296:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017668.660983:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017665/real 1734017668] req@ffff8800a73cce00 x1818247332839424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017668.661000:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332839424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017668.661013:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017668.661017:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017668.661022:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332839424:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734017668.661031:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017665/real 1734017668] req@ffff8800a73ced80 x1818247332839552/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017668.661050:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332839552/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017668.661063:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017668.661066:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017668.661072:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332839552:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017669.042472:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017670.044445:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017670.662331:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017670.662341:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017670.662345:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017670.662356:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1933 00000100:00080000:3.0:1734017670.662365:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017670.662372:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017670.662407:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332840192/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017670.662441:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017670.662445:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017670.662447:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017670.662452:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1933 00000100:00080000:3.0:1734017670.662455:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017670.662459:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017670.662468:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017670.662478:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332840320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017670.662488:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017670.662492:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017670.662494:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017670.662497:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1933 00000100:00080000:3.0:1734017670.662500:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017670.662503:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017670.662508:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017670.662516:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332840448/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017670.662525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017670.662528:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017670.662530:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017670.662533:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1933 00000100:00080000:3.0:1734017670.662536:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017670.662539:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017670.662543:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332840192:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017670.662544:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017670.662551:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332840576/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017670.662560:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017670.662563:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017670.662565:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017670.662568:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1933 00000100:00080000:3.0:1734017670.662570:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017670.662573:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017670.662578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017670.662585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332840704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017670.662593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017670.662631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017670.662633:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017670.662637:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1933 00000100:00080000:3.0:1734017670.662642:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017670.662645:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017670.662650:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017670.662651:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017670.662660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332840832/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017670.662668:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017670.662672:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734017670.662673:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017670/real 1734017670] req@ffff8800a73ced80 x1818247332840192/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017670.662673:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017670.662677:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1933 00000100:00080000:3.0:1734017670.662679:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017670.662683:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017670.662686:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017670.662692:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332840192/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017670.662694:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332840960/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017670.662703:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017670.662706:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017670.662711:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332840192:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017670.662718:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332840320:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017670.662738:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017670.662745:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017670/real 1734017670] req@ffff8800a73cce00 x1818247332840320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017670.662764:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332840320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017670.662772:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017670.662775:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017670.662778:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332840320:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017670.662784:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332840448:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017670.662797:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017670.662803:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017670/real 1734017670] req@ffff8800a73cd500 x1818247332840448/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017670.662813:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332840448/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017670.662820:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017670.662823:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017670.662826:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332840448:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017670.662850:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332840576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017670.662862:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017670.662868:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017670/real 1734017670] req@ffff8800a73cca80 x1818247332840576/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017670.662878:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332840576/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017670.662886:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017670.662889:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017670.662892:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332840576:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017670.662898:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332840704:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017670.662910:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017670.662915:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017670/real 1734017670] req@ffff8800a73cf480 x1818247332840704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017670.662925:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332840704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017670.662933:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017670.662935:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017670.662938:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332840704:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017670.662944:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332840832:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017670.662955:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017670.662961:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017670/real 1734017670] req@ffff8800a73cd880 x1818247332840832/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017670.662970:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332840832/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017670.662978:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017670.662980:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017670.662983:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332840832:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017670.662988:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332840960:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017670.663003:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017670.663009:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017670/real 1734017670] req@ffff8800a73cea00 x1818247332840960/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017670.663019:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332840960/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017670.663026:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017670.663028:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017670.663031:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332840960:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017671.046478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017672.048503:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017673.050497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017674.052510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017675.054484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017675.670386:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017675.670395:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017675.670400:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017675.670411:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1938 00000100:00080000:3.0:1734017675.670422:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017675.670429:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017675.670465:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332841088/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017675.670508:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017675.670551:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017675.670553:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017675.670558:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1938 00000100:00080000:3.0:1734017675.670562:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017675.670566:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017675.670577:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017675.670589:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332841216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017675.670599:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017675.670603:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017675.670604:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017675.670608:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1938 00000100:00080000:3.0:1734017675.670610:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017675.670614:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017675.670619:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017675.670627:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332841344/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017675.670636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017675.670640:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017675.670641:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017675.670643:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332841088:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017675.670645:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1938 00000100:00080000:3.0:1734017675.670647:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017675.670652:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017675.670656:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017675.670665:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332841472/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017675.670673:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017675.670676:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017675.670678:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017675.670681:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1938 00000100:00080000:3.0:1734017675.670684:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017675.670688:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017675.670692:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017675.670700:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332841600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017675.670708:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017675.670711:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017675.670712:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017675.670716:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1938 00000100:00080000:3.0:1734017675.670729:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017675.670732:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734017675.670736:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017675.670737:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017675.670745:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332841728/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017675.670754:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017675.670758:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017675.670759:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017675.670763:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1938 00000100:00100000:0.0:1734017675.670765:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017675/real 1734017675] req@ffff8800a73cc380 x1818247332841088/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017675.670767:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017675.670770:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017675.670774:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017675.670783:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332841856/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017675.670788:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332841088/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017675.670801:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017675.670805:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017675.670810:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332841088:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017675.670819:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332841216:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017675.670839:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017675.670845:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017675/real 1734017675] req@ffff8800a73cf100 x1818247332841216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017675.670857:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332841216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017675.670866:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017675.670870:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017675.670873:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332841216:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017675.670903:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332841344:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017675.670918:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017675.670924:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017675/real 1734017675] req@ffff8800a73cdf80 x1818247332841344/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017675.670935:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332841344/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017675.670943:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017675.670946:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017675.670949:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332841344:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017675.670956:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332841472:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017675.670968:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017675.670974:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017675/real 1734017675] req@ffff8800a73cf800 x1818247332841472/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017675.670984:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332841472/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017675.670991:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017675.670994:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017675.670997:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332841472:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017675.671004:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332841600:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017675.671018:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017675.671024:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017675/real 1734017675] req@ffff8800a73cfb80 x1818247332841600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017675.671035:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332841600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017675.671043:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017675.671045:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017675.671048:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332841600:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017675.671054:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332841728:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017675.671068:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017675.671073:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017675/real 1734017675] req@ffff8800a73cdc00 x1818247332841728/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017675.671083:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332841728/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017675.671090:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017675.671093:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017675.671096:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332841728:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017675.671102:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332841856:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017675.671116:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017675.671122:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017675/real 1734017675] req@ffff8800a73ce680 x1818247332841856/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017675.671134:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332841856/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017675.671142:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017675.671144:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017675.671147:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332841856:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017676.056552:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017677.058535:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017678.060464:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017679.062536:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017680.064524:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017680.678364:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017680.678372:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017680.678377:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017680.678387:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1943 00000100:00080000:3.0:1734017680.678396:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017680.678404:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017680.678466:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332841984/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017680.678497:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017680.678501:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017680.678503:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017680.678508:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1943 00000100:00080000:3.0:1734017680.678511:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017680.678516:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017680.678527:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017680.678540:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332842112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017680.678550:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017680.678554:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017680.678555:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017680.678559:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1943 00000100:00080000:3.0:1734017680.678562:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017680.678565:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017680.678570:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017680.678578:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332842240/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017680.678587:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017680.678590:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017680.678591:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017680.678595:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1943 00000100:00080000:3.0:1734017680.678597:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017680.678600:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017680.678605:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017680.678607:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332841984:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017680.678613:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332842368/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017680.678622:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017680.678625:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017680.678627:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017680.678631:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1943 00000100:00080000:3.0:1734017680.678633:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017680.678637:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017680.678641:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017680.678649:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332842496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017680.678657:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017680.678660:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017680.678662:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017680.678666:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1943 00000100:00080000:3.0:1734017680.678669:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017680.678673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734017680.678674:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017680.678677:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017680.678684:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332842624/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017680.678692:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017680.678695:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734017680.678696:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017680/real 1734017680] req@ffff8800a73cc700 x1818247332841984/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017680.678696:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017680.678700:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1943 00000100:00080000:3.0:1734017680.678702:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017680.678706:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017680.678710:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017680.678715:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332841984/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017680.678719:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332842752/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017680.678726:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017680.678730:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017680.678734:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332841984:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017680.678742:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332842112:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017680.678760:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017680.678766:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017680/real 1734017680] req@ffff8800a73ce680 x1818247332842112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017680.678777:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332842112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017680.678785:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017680.678788:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017680.678792:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332842112:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017680.678798:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332842240:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017680.678812:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017680.678818:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017680/real 1734017680] req@ffff8800a73cdc00 x1818247332842240/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017680.678828:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332842240/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017680.678836:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017680.678838:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017680.678842:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332842240:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017680.678866:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332842368:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017680.678878:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017680.678884:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017680/real 1734017680] req@ffff8800a73cfb80 x1818247332842368/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017680.678894:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332842368/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017680.678902:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017680.678905:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017680.678907:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332842368:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017680.678913:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332842496:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017680.678928:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017680.678934:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017680/real 1734017680] req@ffff8800a73cf800 x1818247332842496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017680.678944:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332842496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017680.678952:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017680.678954:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017680.678957:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332842496:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017680.678962:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332842624:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017680.678974:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017680.678980:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017680/real 1734017680] req@ffff8800a73cdf80 x1818247332842624/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017680.678989:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332842624/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017680.678996:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017680.678998:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017680.679002:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332842624:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017680.679010:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332842752:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017680.679032:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017680.679044:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017680/real 1734017680] req@ffff8800a73cf100 x1818247332842752/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017680.679069:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332842752/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017680.679082:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017680.679086:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017680.679091:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332842752:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017681.066497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017682.068500:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017683.070552:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017684.072456:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017685.074506:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017685.686397:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017685.686406:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017685.686410:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017685.686422:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1948 00000100:00080000:3.0:1734017685.686432:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017685.686440:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017685.686474:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332842880/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017685.686506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017685.686511:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017685.686513:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017685.686517:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1948 00000100:00080000:3.0:1734017685.686520:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017685.686524:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017685.686534:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017685.686544:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332843008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017685.686554:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017685.686557:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017685.686559:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017685.686563:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1948 00000100:00080000:3.0:1734017685.686567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017685.686571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017685.686575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017685.686583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332843136/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017685.686592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017685.686595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017685.686596:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017685.686599:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1948 00000100:00080000:3.0:1734017685.686602:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017685.686605:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017685.686609:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017685.686617:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332843264/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017685.686621:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332842880:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017685.686626:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017685.686629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017685.686631:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017685.686634:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1948 00000100:00080000:3.0:1734017685.686637:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017685.686640:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017685.686644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017685.686652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332843392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017685.686667:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017685.686670:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017685.686671:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017685.686675:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1948 00000100:00080000:3.0:1734017685.686679:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017685.686682:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017685.686686:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017685.686688:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017685.686693:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332843520/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017685.686701:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017685.686704:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017685.686706:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017685.686709:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017685/real 1734017685] req@ffff8800a73cc380 x1818247332842880/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017685.686709:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1948 00000100:00080000:3.0:1734017685.686712:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017685.686715:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017685.686719:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017685.686728:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332843648/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017685.686729:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332842880/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017685.686740:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017685.686744:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017685.686748:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332842880:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017685.686756:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332843008:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017685.686773:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017685.686780:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017685/real 1734017685] req@ffff8800a73cea00 x1818247332843008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017685.686792:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332843008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017685.686799:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017685.686803:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017685.686807:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332843008:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017685.686813:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332843136:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017685.686827:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017685.686833:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017685/real 1734017685] req@ffff8800a73cd880 x1818247332843136/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017685.686843:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332843136/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017685.686851:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017685.686853:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017685.686857:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332843136:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017685.686881:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332843264:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017685.686893:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017685.686899:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017685/real 1734017685] req@ffff8800a73cf480 x1818247332843264/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017685.686910:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332843264/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017685.686918:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017685.686920:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017685.686923:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332843264:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017685.686929:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332843392:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017685.686944:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017685.686950:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017685/real 1734017685] req@ffff8800a73cca80 x1818247332843392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017685.686960:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332843392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017685.686968:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017685.686970:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017685.686973:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332843392:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017685.686979:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332843520:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017685.686990:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017685.686996:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017685/real 1734017685] req@ffff8800a73cd500 x1818247332843520/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017685.687006:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332843520/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017685.687013:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017685.687015:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017685.687018:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332843520:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017685.687023:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332843648:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017685.687035:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017685.687040:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017685/real 1734017685] req@ffff8800a73cce00 x1818247332843648/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017685.687057:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332843648/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017685.687065:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017685.687067:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017685.687070:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332843648:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017686.076480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017687.078522:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017688.080510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017689.082515:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017690.084495:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017690.694364:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017690.694373:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017690.694379:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017690.694389:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1953 00000100:00080000:3.0:1734017690.694399:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017690.694406:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017690.694442:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332843776/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017690.694475:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017690.694479:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017690.694481:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017690.694485:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1953 00000100:00080000:3.0:1734017690.694489:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017690.694493:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017690.694502:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017690.694515:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332843904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017690.694525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017690.694528:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017690.694530:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017690.694533:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1953 00000100:00080000:3.0:1734017690.694536:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017690.694540:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017690.694544:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017690.694553:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332844032/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017690.694568:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017690.694572:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017690.694573:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017690.694576:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1953 00000100:00080000:3.0:1734017690.694579:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017690.694582:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017690.694585:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332843776:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017690.694587:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017690.694594:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332844160/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017690.694603:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017690.694606:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017690.694608:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017690.694611:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1953 00000100:00080000:3.0:1734017690.694613:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017690.694616:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017690.694621:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017690.694628:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332844288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017690.694636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017690.694639:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017690.694641:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017690.694645:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1953 00000100:00080000:3.0:1734017690.694648:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734017690.694651:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017690.694652:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017690.694656:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017690.694663:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332844416/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017690.694671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017690.694674:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017690/real 1734017690] req@ffff8800a73ced80 x1818247332843776/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017690.694674:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017690.694676:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017690.694679:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1953 00000100:00080000:3.0:1734017690.694682:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017690.694685:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017690.694689:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017690.694694:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332843776/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017690.694697:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332844544/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017690.694705:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017690.694708:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017690.694713:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332843776:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017690.694721:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332843904:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017690.694738:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017690.694745:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017690/real 1734017690] req@ffff8800a73cce00 x1818247332843904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017690.694756:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332843904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017690.694764:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017690.694767:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017690.694770:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332843904:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017690.694777:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332844032:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017690.694790:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017690.694796:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017690/real 1734017690] req@ffff8800a73cd500 x1818247332844032/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017690.694806:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332844032/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017690.694814:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017690.694816:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017690.694820:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332844032:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017690.694844:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332844160:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017690.694856:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017690.694862:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017690/real 1734017690] req@ffff8800a73cca80 x1818247332844160/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017690.694872:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332844160/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017690.694880:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017690.694882:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017690.694885:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332844160:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017690.694891:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332844288:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017690.694906:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017690.694912:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017690/real 1734017690] req@ffff8800a73cf480 x1818247332844288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017690.694922:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332844288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017690.694930:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017690.694932:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017690.694935:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332844288:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017690.694941:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332844416:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017690.694952:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017690.694958:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017690/real 1734017690] req@ffff8800a73cd880 x1818247332844416/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017690.694968:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332844416/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017690.694975:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017690.694977:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017690.694980:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332844416:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017690.694986:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332844544:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017690.694998:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017690.695004:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017690/real 1734017690] req@ffff8800a73cea00 x1818247332844544/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017690.695020:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332844544/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017690.695027:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017690.695029:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017690.695032:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332844544:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017691.086490:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017692.088504:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017693.090488:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017694.092507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017695.094456:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017695.702329:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017695.702338:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017695.702343:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017695.702353:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1958 00000100:00080000:3.0:1734017695.702363:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017695.702370:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017695.702405:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332844672/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017695.702445:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017695.702450:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017695.702452:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017695.702456:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1958 00000100:00080000:3.0:1734017695.702459:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017695.702463:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017695.702473:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017695.702483:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332844800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017695.702492:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017695.702496:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017695.702497:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017695.702501:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1958 00000100:00080000:3.0:1734017695.702503:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017695.702507:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017695.702512:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017695.702519:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332844928/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017695.702527:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017695.702530:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017695.702532:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017695.702535:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1958 00000100:00080000:3.0:1734017695.702538:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017695.702542:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017695.702547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017695.702548:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332844672:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017695.702554:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332845056/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017695.702563:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017695.702567:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017695.702568:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017695.702571:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1958 00000100:00080000:3.0:1734017695.702574:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017695.702577:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017695.702581:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017695.702589:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332845184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017695.702597:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017695.702600:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017695.702602:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017695.702605:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1958 00000100:00080000:3.0:1734017695.702608:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734017695.702612:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017695.702612:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017695.702616:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017695.702623:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332845312/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017695.702632:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017695.702633:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017695/real 1734017695] req@ffff8800a73cc380 x1818247332844672/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017695.702636:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017695.702637:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017695.702641:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1958 00000100:00080000:3.0:1734017695.702644:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017695.702647:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017695.702651:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017695.702653:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332844672/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017695.702660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332845440/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017695.702663:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017695.702666:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017695.702671:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332844672:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017695.702679:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332844800:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017695.702695:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017695.702702:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017695/real 1734017695] req@ffff8800a73cf100 x1818247332844800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017695.702713:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332844800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017695.702721:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017695.702724:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017695.702727:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332844800:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017695.702734:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332844928:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017695.702747:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017695.702753:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017695/real 1734017695] req@ffff8800a73cdf80 x1818247332844928/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017695.702763:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332844928/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017695.702771:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017695.702773:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017695.702777:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332844928:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017695.702800:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332845056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017695.702812:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017695.702819:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017695/real 1734017695] req@ffff8800a73cf800 x1818247332845056/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017695.702829:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332845056/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017695.702837:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017695.702839:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017695.702842:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332845056:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017695.702847:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332845184:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017695.702862:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017695.702868:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017695/real 1734017695] req@ffff8800a73cfb80 x1818247332845184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017695.702879:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332845184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017695.702886:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017695.702889:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017695.702892:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332845184:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017695.702898:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332845312:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017695.702909:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017695.702915:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017695/real 1734017695] req@ffff8800a73cdc00 x1818247332845312/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017695.702925:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332845312/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017695.702932:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017695.702934:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017695.702937:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332845312:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017695.702942:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332845440:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017695.702954:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017695.702960:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017695/real 1734017695] req@ffff8800a73ce680 x1818247332845440/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017695.702976:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332845440/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017695.702984:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017695.702987:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017695.702989:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332845440:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017696.096477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017697.098468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017698.100485:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017699.102527:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017700.104494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017700.710370:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017700.710380:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017700.710385:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017700.710396:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1963 00000100:00080000:3.0:1734017700.710406:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017700.710414:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017700.710452:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332845568/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017700.710489:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017700.710493:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017700.710495:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017700.710500:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1963 00000100:00080000:3.0:1734017700.710503:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017700.710507:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017700.710518:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017700.710531:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332845696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017700.710542:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017700.710545:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017700.710547:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017700.710550:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1963 00000100:00080000:3.0:1734017700.710553:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017700.710557:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017700.710561:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017700.710570:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332845824/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017700.710579:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017700.710582:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017700.710583:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017700.710587:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1963 00000100:00080000:3.0:1734017700.710589:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017700.710593:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017700.710597:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017700.710605:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332845952/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017700.710614:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017700.710618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017700.710620:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017700.710623:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332845568:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017700.710624:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1963 00000100:00080000:3.0:1734017700.710627:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017700.710631:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017700.710635:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017700.710644:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332846080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017700.710652:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017700.710655:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017700.710656:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017700.710660:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1963 00000100:00080000:3.0:1734017700.710663:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017700.710666:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017700.710671:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017700.710678:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332846208/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017700.710685:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017700.710688:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017700.710690:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017700.710693:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1963 00000100:00080000:3.0:1734017700.710696:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017700.710699:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017700.710703:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017700.710711:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332846336/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734017700.710715:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017700.710744:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017700/real 1734017700] req@ffff8800a73cc700 x1818247332845568/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017700.710767:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332845568/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017700.710780:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017700.710784:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017700.710790:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332845568:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017700.710798:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332845696:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017700.710819:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017700.710826:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017700/real 1734017700] req@ffff8800a73ce680 x1818247332845696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017700.710837:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332845696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017700.710845:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017700.710850:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017700.710853:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332845696:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017700.710861:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332845824:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017700.710876:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017700.710882:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017700/real 1734017700] req@ffff8800a73cdc00 x1818247332845824/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017700.710892:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332845824/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017700.710900:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017700.710903:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017700.710907:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332845824:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017700.710937:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332845952:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017700.710950:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017700.710957:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017700/real 1734017700] req@ffff8800a73cfb80 x1818247332845952/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017700.710968:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332845952/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017700.710976:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017700.710978:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017700.710982:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332845952:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017700.710987:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332846080:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017700.711004:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017700.711010:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017700/real 1734017700] req@ffff8800a73cf800 x1818247332846080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017700.711021:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332846080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017700.711028:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017700.711030:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017700.711033:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332846080:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017700.711039:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332846208:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017700.711052:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017700.711057:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017700/real 1734017700] req@ffff8800a73cdf80 x1818247332846208/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017700.711068:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332846208/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017700.711076:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017700.711078:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017700.711081:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332846208:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017700.711132:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332846336:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017700.711150:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017700.711158:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017700/real 1734017700] req@ffff8800a73cf100 x1818247332846336/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017700.711175:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332846336/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017700.711183:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017700.711186:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017700.711189:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332846336:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017701.106500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017702.108468:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017703.110511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017704.112478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017705.114511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017705.718359:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017705.718369:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017705.718373:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017705.718383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1968 00000100:00080000:3.0:1734017705.718394:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017705.718402:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017705.718440:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332846464/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017705.718473:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017705.718477:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017705.718479:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017705.718484:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1968 00000100:00080000:3.0:1734017705.718487:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017705.718491:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017705.718500:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017705.718510:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332846592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017705.718520:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017705.718524:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017705.718525:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017705.718529:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1968 00000100:00080000:3.0:1734017705.718534:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017705.718537:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017705.718542:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017705.718550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332846720/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017705.718558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017705.718561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017705.718563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017705.718566:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1968 00000100:00080000:3.0:1734017705.718568:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017705.718572:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017705.718576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017705.718579:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332846464:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017705.718584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332846848/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017705.718593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017705.718596:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017705.718597:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017705.718601:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1968 00000100:00080000:3.0:1734017705.718604:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017705.718607:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017705.718611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017705.718619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332846976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017705.718633:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017705.718636:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017705.718638:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017705.718642:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1968 00000100:00080000:3.0:1734017705.718645:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734017705.718647:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017705.718648:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017705.718652:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017705.718660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332847104/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017705.718667:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017705.718668:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017705/real 1734017705] req@ffff8800a73cc380 x1818247332846464/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017705.718671:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017705.718672:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017705.718676:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1968 00000100:00080000:3.0:1734017705.718678:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017705.718681:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017705.718686:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017705.718687:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332846464/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017705.718695:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332847232/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017705.718698:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017705.718702:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017705.718707:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332846464:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017705.718714:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332846592:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017705.718732:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017705.718738:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017705/real 1734017705] req@ffff8800a73cea00 x1818247332846592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017705.718749:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332846592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017705.718758:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017705.718761:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017705.718765:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332846592:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017705.718772:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332846720:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017705.718786:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017705.718792:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017705/real 1734017705] req@ffff8800a73cd880 x1818247332846720/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017705.718802:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332846720/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017705.718810:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017705.718813:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017705.718816:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332846720:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017705.718840:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332846848:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017705.718853:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017705.718859:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017705/real 1734017705] req@ffff8800a73cf480 x1818247332846848/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017705.718869:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332846848/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017705.718882:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017705.718885:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017705.718888:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332846848:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017705.718893:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332846976:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017705.718908:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017705.718914:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017705/real 1734017705] req@ffff8800a73cca80 x1818247332846976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017705.718925:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332846976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017705.718932:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017705.718935:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017705.718938:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332846976:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017705.718944:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332847104:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017705.718957:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017705.718962:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017705/real 1734017705] req@ffff8800a73cd500 x1818247332847104/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017705.718972:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332847104/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017705.718980:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017705.718982:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017705.718985:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332847104:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017705.718991:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332847232:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017705.719041:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017705.719050:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017705/real 1734017705] req@ffff8800a73cce00 x1818247332847232/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017705.719067:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332847232/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017705.719076:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017705.719078:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017705.719081:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332847232:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017706.116451:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017707.118494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017708.120465:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017709.122497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017710.124471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017710.726374:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017710.726383:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017710.726388:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017710.726398:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1973 00000100:00080000:3.0:1734017710.726408:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017710.726416:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017710.726451:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332847360/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017710.726487:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017710.726491:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017710.726493:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017710.726498:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1973 00000100:00080000:3.0:1734017710.726501:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017710.726505:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017710.726515:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017710.726526:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332847488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017710.726544:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017710.726547:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017710.726549:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017710.726553:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1973 00000100:00080000:3.0:1734017710.726556:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017710.726559:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017710.726564:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017710.726572:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332847616/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017710.726588:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017710.726591:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017710.726593:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017710.726596:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1973 00000100:00080000:3.0:1734017710.726599:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017710.726602:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017710.726606:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332847360:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017710.726607:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017710.726615:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332847744/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017710.726623:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017710.726627:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017710.726628:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017710.726631:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1973 00000100:00080000:3.0:1734017710.726634:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017710.726637:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017710.726641:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017710.726649:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332847872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017710.726657:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017710.726660:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017710.726662:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017710.726666:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1973 00000100:00080000:3.0:1734017710.726670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734017710.726672:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017710.726674:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017710.726678:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017710.726686:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332848000/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017710.726694:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017710/real 1734017710] req@ffff8800a73ced80 x1818247332847360/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017710.726694:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017710.726697:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017710.726698:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017710.726701:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1973 00000100:00080000:3.0:1734017710.726704:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017710.726707:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017710.726711:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017710.726713:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332847360/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017710.726719:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332848128/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017710.726724:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017710.726728:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017710.726733:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332847360:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017710.726740:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332847488:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017710.726757:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017710.726764:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017710/real 1734017710] req@ffff8800a73cce00 x1818247332847488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017710.726775:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332847488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017710.726783:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017710.726787:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017710.726790:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332847488:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017710.726797:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332847616:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017710.726811:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017710.726817:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017710/real 1734017710] req@ffff8800a73cd500 x1818247332847616/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017710.726827:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332847616/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017710.726835:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017710.726838:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017710.726842:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332847616:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017710.726867:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332847744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017710.726879:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017710.726885:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017710/real 1734017710] req@ffff8800a73cca80 x1818247332847744/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017710.726896:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332847744/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017710.726905:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017710.726907:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017710.726909:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332847744:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017710.726951:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332847872:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017710.726972:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017710.726980:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017710/real 1734017710] req@ffff8800a73cf480 x1818247332847872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017710.726992:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332847872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017710.727000:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017710.727002:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017710.727005:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332847872:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017710.727011:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332848000:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017710.727023:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017710.727029:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017710/real 1734017710] req@ffff8800a73cd880 x1818247332848000/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017710.727038:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332848000/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017710.727046:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017710.727048:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017710.727052:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332848000:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017710.727057:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332848128:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017710.727069:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017710.727074:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017710/real 1734017710] req@ffff8800a73cea00 x1818247332848128/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017710.727091:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332848128/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017710.727099:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017710.727101:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017710.727104:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332848128:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017711.126484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017712.128500:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017713.130507:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017714.132477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017715.134457:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017715.734360:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017715.734369:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017715.734374:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017715.734383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1978 00000100:00080000:3.0:1734017715.734393:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017715.734401:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017715.734436:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332848256/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017715.734478:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017715.734482:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017715.734484:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017715.734488:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1978 00000100:00080000:3.0:1734017715.734491:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017715.734495:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017715.734505:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017715.734516:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332848384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017715.734525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017715.734528:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017715.734530:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017715.734534:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1978 00000100:00080000:3.0:1734017715.734536:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017715.734540:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017715.734544:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017715.734552:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332848512/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017715.734560:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017715.734563:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017715.734565:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017715.734568:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1978 00000100:00080000:3.0:1734017715.734570:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017715.734574:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017715.734579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017715.734586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332848640/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017715.734594:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332848256:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017715.734595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017715.734598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017715.734600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017715.734603:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1978 00000100:00080000:3.0:1734017715.734606:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017715.734609:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017715.734613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017715.734620:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332848768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017715.734628:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017715.734631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017715.734632:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017715.734636:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1978 00000100:00080000:3.0:1734017715.734639:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017715.734643:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017715.734647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017715.734655:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332848896/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734017715.734662:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017715.734663:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017715.734667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017715.734668:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017715.734672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1978 00000100:00100000:0.0:1734017715.734684:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017715/real 1734017715] req@ffff8800a73cc380 x1818247332848256/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017715.734685:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017715.734689:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017715.734693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017715.734702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332849024/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017715.734703:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332848256/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017715.734715:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017715.734718:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017715.734723:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332848256:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017715.734730:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332848384:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017715.734748:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017715.734754:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017715/real 1734017715] req@ffff8800a73cf100 x1818247332848384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017715.734766:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332848384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017715.734774:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017715.734778:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017715.734781:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332848384:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017715.734787:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332848512:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017715.734801:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017715.734807:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017715/real 1734017715] req@ffff8800a73cdf80 x1818247332848512/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017715.734817:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332848512/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017715.734825:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017715.734828:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017715.734876:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332848512:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017715.734902:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332848640:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017715.734920:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017715.734927:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017715/real 1734017715] req@ffff8800a73cf800 x1818247332848640/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017715.734939:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332848640/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017715.734947:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017715.734949:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017715.734952:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332848640:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017715.734957:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332848768:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017715.734973:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017715.734979:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017715/real 1734017715] req@ffff8800a73cfb80 x1818247332848768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017715.734990:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332848768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017715.734997:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017715.734999:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017715.735002:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332848768:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017715.735007:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332848896:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017715.735019:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017715.735024:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017715/real 1734017715] req@ffff8800a73cdc00 x1818247332848896/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017715.735034:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332848896/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017715.735041:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017715.735043:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017715.735046:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332848896:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017715.735051:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332849024:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017715.735063:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017715.735069:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017715/real 1734017715] req@ffff8800a73ce680 x1818247332849024/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017715.735086:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332849024/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017715.735094:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017715.735096:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017715.735099:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332849024:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017716.136499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017717.138515:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017718.140486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017719.142478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017720.144488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017720.742364:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017720.742373:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017720.742378:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017720.742389:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1983 00000100:00080000:3.0:1734017720.742398:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017720.742406:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017720.742442:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332849152/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017720.742475:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017720.742479:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017720.742481:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017720.742486:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1983 00000100:00080000:3.0:1734017720.742489:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017720.742493:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017720.742503:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017720.742515:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332849280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017720.742525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017720.742529:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017720.742530:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017720.742534:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1983 00000100:00080000:3.0:1734017720.742536:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017720.742540:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017720.742544:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017720.742552:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332849408/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017720.742561:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017720.742564:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017720.742565:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017720.742568:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1983 00000100:00080000:3.0:1734017720.742571:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017720.742574:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017720.742578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017720.742581:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332849152:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017720.742586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332849536/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017720.742595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017720.742598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017720.742601:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017720.742604:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1983 00000100:00080000:3.0:1734017720.742607:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017720.742610:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017720.742615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017720.742622:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332849664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017720.742630:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017720.742633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017720.742635:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017720.742639:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1983 00000100:00080000:3.0:1734017720.742641:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017720.742645:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734017720.742648:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017720.742649:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017720.742656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332849792/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017720.742664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017720.742667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017720.742668:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017720.742670:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017720/real 1734017720] req@ffff8800a73cc700 x1818247332849152/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017720.742672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1983 00000100:00080000:3.0:1734017720.742675:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017720.742678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017720.742682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017720.742690:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332849152/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017720.742690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332849920/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017720.742700:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017720.742704:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017720.742709:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332849152:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017720.742716:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332849280:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017720.742734:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017720.742741:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017720/real 1734017720] req@ffff8800a73ce680 x1818247332849280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017720.742787:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332849280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017720.742796:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017720.742800:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017720.742803:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332849280:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017720.742810:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332849408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017720.742827:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017720.742834:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017720/real 1734017720] req@ffff8800a73cdc00 x1818247332849408/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017720.742845:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332849408/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017720.742853:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017720.742855:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017720.742858:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332849408:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017720.742882:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332849536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017720.742894:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017720.742901:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017720/real 1734017720] req@ffff8800a73cfb80 x1818247332849536/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017720.742912:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332849536/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017720.742919:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017720.742921:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017720.742924:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332849536:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017720.742930:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332849664:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017720.742945:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017720.742951:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017720/real 1734017720] req@ffff8800a73cf800 x1818247332849664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017720.742961:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332849664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017720.742969:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017720.742971:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017720.742974:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332849664:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017720.742979:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332849792:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017720.742991:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017720.742997:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017720/real 1734017720] req@ffff8800a73cdf80 x1818247332849792/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017720.743007:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332849792/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017720.743015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017720.743017:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017720.743020:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332849792:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017720.743025:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332849920:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017720.743037:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017720.743042:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017720/real 1734017720] req@ffff8800a73cf100 x1818247332849920/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017720.743058:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332849920/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017720.743066:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017720.743068:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017720.743071:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332849920:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017721.146481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017722.148483:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017723.150480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017724.152496:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017725.154467:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017725.750365:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017725.750374:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017725.750379:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017725.750389:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1988 00000100:00080000:3.0:1734017725.750399:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017725.750406:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017725.750441:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332850048/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017725.750475:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017725.750479:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017725.750481:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017725.750486:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1988 00000100:00080000:3.0:1734017725.750489:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017725.750493:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017725.750502:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017725.750512:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332850176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017725.750522:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017725.750525:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017725.750527:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017725.750531:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1988 00000100:00080000:3.0:1734017725.750536:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017725.750539:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017725.750545:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017725.750552:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332850304/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017725.750561:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017725.750564:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017725.750565:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017725.750569:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1988 00000100:00080000:3.0:1734017725.750571:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017725.750575:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017725.750579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017725.750580:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332850048:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017725.750586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332850432/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017725.750595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017725.750598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017725.750600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017725.750603:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1988 00000100:00080000:3.0:1734017725.750605:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017725.750609:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017725.750613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017725.750621:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332850560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017725.750635:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017725.750639:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017725.750640:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017725.750644:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1988 00000800:00000100:0.0:1734017725.750645:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017725.750647:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017725.750650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017725.750654:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017725.750685:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332850688/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017725.750693:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017725/real 1734017725] req@ffff8800a73cc380 x1818247332850048/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017725.750694:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017725.750697:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017725.750699:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017725.750702:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1988 00000100:00080000:3.0:1734017725.750705:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017725.750708:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017725.750713:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332850048/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017725.750713:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017725.750723:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332850816/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017725.750724:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017725.750727:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017725.750732:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332850048:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017725.750740:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332850176:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017725.750759:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017725.750766:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017725/real 1734017725] req@ffff8800a73cea00 x1818247332850176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017725.750777:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332850176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017725.750786:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017725.750789:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017725.750793:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332850176:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017725.750799:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332850304:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017725.750813:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017725.750819:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017725/real 1734017725] req@ffff8800a73cd880 x1818247332850304/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017725.750829:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332850304/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017725.750837:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017725.750839:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017725.750843:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332850304:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017725.750866:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332850432:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017725.750879:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017725.750885:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017725/real 1734017725] req@ffff8800a73cf480 x1818247332850432/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017725.750896:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332850432/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017725.750903:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017725.750905:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017725.750909:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332850432:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017725.750914:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332850560:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017725.750929:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017725.750935:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017725/real 1734017725] req@ffff8800a73cca80 x1818247332850560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017725.750945:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332850560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017725.750953:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017725.750955:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017725.750958:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332850560:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017725.750963:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332850688:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017725.750974:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017725.750980:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017725/real 1734017725] req@ffff8800a73cd500 x1818247332850688/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017725.750990:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332850688/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017725.750998:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017725.751000:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017725.751003:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332850688:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017725.751009:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332850816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017725.751020:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017725.751026:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017725/real 1734017725] req@ffff8800a73cce00 x1818247332850816/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017725.751043:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332850816/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017725.751051:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017725.751053:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017725.751056:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332850816:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017726.156457:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017727.158484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017728.160462:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017729.162483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017730.164485:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017730.758332:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017730.758341:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017730.758347:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017730.758357:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1993 00000100:00080000:3.0:1734017730.758366:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017730.758374:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017730.758410:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332850944/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017730.758444:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017730.758448:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017730.758450:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017730.758454:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1993 00000100:00080000:3.0:1734017730.758458:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017730.758461:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017730.758470:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017730.758483:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332851072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017730.758493:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017730.758496:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017730.758497:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017730.758501:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1993 00000100:00080000:3.0:1734017730.758504:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017730.758507:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017730.758512:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017730.758520:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332851200/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017730.758537:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017730.758540:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017730.758542:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017730.758545:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332850944:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017730.758545:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1993 00000100:00080000:3.0:1734017730.758548:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017730.758551:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017730.758555:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017730.758563:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332851328/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017730.758571:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017730.758608:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017730.758610:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017730.758614:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1993 00000100:00080000:3.0:1734017730.758617:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017730.758620:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017730.758625:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017730.758634:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332851456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017730.758643:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017730.758646:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017730.758647:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017730.758651:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1993 00000100:00080000:3.0:1734017730.758655:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017730.758658:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017730.758662:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017730.758670:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332851584/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017730.758674:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332851072:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017730.758677:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017730.758680:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017730.758682:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017730.758685:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1993 00000100:00080000:3.0:1734017730.758689:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017730.758693:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017730.758697:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017730.758703:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332851200:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734017730.758705:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332851712/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017730.758719:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332851328:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017730.758733:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332851456:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017730.758747:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332851584:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017730.758763:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332851712:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017731.166455:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017732.168492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017733.170448:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1734017733.764488:0:1955:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:1.0:1734017733.764561:0:1955:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1734017733.764581:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017733.764619:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017733.764627:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017733.764634:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017733.764640:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017733.764647:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017733.764653:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:0.0:1734017733.764745:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017730/real 1734017733] req@ffff8800a73cea00 x1818247332851712/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017733.764783:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332851712/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017733.764801:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017733.764807:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017733.764815:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332851712:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017733.764828:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017730/real 1734017733] req@ffff8800a73cd500 x1818247332851200/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017733.764847:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332851200/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017733.764860:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017733.764864:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017733.764870:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332851200:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017733.764881:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017730/real 1734017733] req@ffff8800a73cca80 x1818247332851328/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017733.764912:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332851328/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017733.764924:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017733.764928:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017733.764933:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332851328:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017733.764943:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017730/real 1734017733] req@ffff8800a73cf480 x1818247332851456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017733.764961:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332851456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017733.764973:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017733.764978:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017733.764983:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332851456:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017733.764997:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017730/real 1734017733] req@ffff8800a73cd880 x1818247332851584/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017733.765015:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332851584/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017733.765027:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017733.765031:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017733.765037:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332851584:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017733.765047:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017730/real 1734017733] req@ffff8800a73ced80 x1818247332850944/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017733.765067:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332850944/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017733.765079:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017733.765083:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017733.765088:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332850944:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017733.765098:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017730/real 1734017733] req@ffff8800a73cce00 x1818247332851072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017733.765116:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332851072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017733.765137:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017733.765141:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017733.765147:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332851072:192.168.201.152@tcp:38:kworker.0 00010000:00010000:3.0:1734017734.172451:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017735.174483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017735.766341:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017735.766350:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017735.766355:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017735.766365:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 1998 00000100:00080000:3.0:1734017735.766374:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017735.766382:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017735.766416:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332851840/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017735.766456:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017735.766460:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017735.766462:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017735.766467:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1998 00000100:00080000:3.0:1734017735.766470:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017735.766474:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017735.766484:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017735.766527:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332851968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017735.766537:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017735.766541:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017735.766542:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017735.766546:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1998 00000100:00080000:3.0:1734017735.766549:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017735.766553:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017735.766558:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734017735.766559:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332851840:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017735.766567:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332852096/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017735.766576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017735.766579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017735.766581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017735.766585:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 1998 00000100:00080000:3.0:1734017735.766587:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017735.766591:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017735.766596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017735.766604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332852224/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017735.766612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017735.766615:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017735.766616:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017735.766620:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1998 00000100:00080000:3.0:1734017735.766623:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734017735.766624:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017735.766626:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017735.766630:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017735.766638:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332852352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017735.766644:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017735/real 1734017735] req@ffff8800a73cc380 x1818247332851840/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017735.766646:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017735.766649:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017735.766650:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017735.766654:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1998 00000100:00080000:3.0:1734017735.766657:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017735.766660:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017735.766663:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332851840/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017735.766664:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017735.766672:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332852480/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017735.766673:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017735.766677:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734017735.766680:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017735.766682:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332851840:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017735.766684:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017735.766686:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017735.766689:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332851968:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017735.766689:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 1998 00000100:00080000:3.0:1734017735.766692:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017735.766695:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017735.766699:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017735.766706:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017735.766707:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332852608/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017735.766714:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017735/real 1734017735] req@ffff8800a73cf100 x1818247332851968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017735.766725:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332851968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017735.766733:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017735.766735:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017735.766739:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332851968:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017735.766760:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332852096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017735.766774:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017735.766781:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017735/real 1734017735] req@ffff8800a73cdf80 x1818247332852096/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017735.766791:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332852096/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017735.766799:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017735.766802:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017735.766805:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332852096:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017735.766810:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332852224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017735.766822:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017735.766827:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017735/real 1734017735] req@ffff8800a73cf800 x1818247332852224/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017735.766837:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332852224/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017735.766844:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017735.766846:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017735.766850:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332852224:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017735.766855:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332852352:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017735.766867:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017735.766872:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017735/real 1734017735] req@ffff8800a73cfb80 x1818247332852352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017735.766882:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332852352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017735.766890:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017735.766892:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017735.766895:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332852352:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017735.766902:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332852480:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017735.766914:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017735.766920:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017735/real 1734017735] req@ffff8800a73cdc00 x1818247332852480/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017735.766929:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332852480/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017735.766937:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017735.766939:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017735.766942:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332852480:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017735.766948:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332852608:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017735.766959:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017735.766965:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017735/real 1734017735] req@ffff8800a73ce680 x1818247332852608/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017735.766975:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332852608/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017735.766982:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017735.766985:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017735.766988:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332852608:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017736.176476:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017737.178522:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017738.180516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017739.182503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017740.184509:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017740.774375:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017740.774385:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017740.774389:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017740.774400:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2003 00000100:00080000:3.0:1734017740.774444:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017740.774453:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017740.774488:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332852736/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017740.774520:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017740.774524:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017740.774526:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017740.774531:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2003 00000100:00080000:3.0:1734017740.774534:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017740.774538:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017740.774549:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017740.774561:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332852864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017740.774571:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017740.774575:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017740.774577:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017740.774580:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2003 00000100:00080000:3.0:1734017740.774582:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017740.774586:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017740.774591:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017740.774599:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332852992/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017740.774607:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017740.774611:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017740.774612:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017740.774615:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2003 00000100:00080000:3.0:1734017740.774618:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017740.774621:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017740.774626:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017740.774631:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332852736:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017740.774634:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332853120/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017740.774642:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017740.774646:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017740.774648:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017740.774652:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2003 00000100:00080000:3.0:1734017740.774654:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017740.774657:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017740.774662:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017740.774670:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332853248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017740.774678:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017740.774681:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017740.774682:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017740.774686:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2003 00000100:00080000:3.0:1734017740.774689:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017740.774692:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734017740.774696:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017740.774696:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017740.774703:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332853376/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017740.774711:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017740.774714:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017740.774716:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017740.774718:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017740/real 1734017740] req@ffff8800a73cc700 x1818247332852736/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017740.774719:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2003 00000100:00080000:3.0:1734017740.774722:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017740.774725:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017740.774729:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017740.774737:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332853504/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017740.774739:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332852736/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017740.774751:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017740.774754:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017740.774758:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332852736:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017740.774766:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332852864:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017740.774784:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017740.774790:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017740/real 1734017740] req@ffff8800a73ce680 x1818247332852864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017740.774802:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332852864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017740.774809:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017740.774812:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017740.774815:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332852864:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017740.774821:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332852992:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017740.774834:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017740.774840:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017740/real 1734017740] req@ffff8800a73cdc00 x1818247332852992/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017740.774850:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332852992/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017740.774857:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017740.774860:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017740.774863:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332852992:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017740.774887:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332853120:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017740.774900:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017740.774906:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017740/real 1734017740] req@ffff8800a73cfb80 x1818247332853120/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017740.774916:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332853120/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017740.774924:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017740.774926:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017740.774929:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332853120:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017740.774934:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332853248:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017740.774946:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017740.774952:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017740/real 1734017740] req@ffff8800a73cf800 x1818247332853248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017740.774961:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332853248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017740.774969:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017740.774971:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017740.774974:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332853248:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017740.774979:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332853376:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017740.774994:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017740.775000:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017740/real 1734017740] req@ffff8800a73cdf80 x1818247332853376/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017740.775010:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332853376/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017740.775017:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017740.775019:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017740.775022:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332853376:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017740.775028:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332853504:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017740.775039:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017740.775045:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017740/real 1734017740] req@ffff8800a73cf100 x1818247332853504/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017740.775054:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332853504/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017740.775062:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017740.775064:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017740.775067:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332853504:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017741.186484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017742.188462:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017743.190524:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017744.192533:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017745.194517:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017745.782394:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017745.782403:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017745.782408:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017745.782418:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2008 00000100:00080000:3.0:1734017745.782428:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017745.782436:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017745.782471:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332853632/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017745.782504:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017745.782508:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017745.782510:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017745.782515:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2008 00000100:00080000:3.0:1734017745.782518:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017745.782522:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017745.782531:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017745.782542:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332853760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017745.782552:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017745.782555:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017745.782557:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017745.782560:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2008 00000100:00080000:3.0:1734017745.782565:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017745.782569:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017745.782573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017745.782582:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332853888/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017745.782590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017745.782593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017745.782595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017745.782598:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2008 00000100:00080000:3.0:1734017745.782600:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017745.782604:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017745.782608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017745.782614:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332853632:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017745.782618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332854016/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017745.782632:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017745.782637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017745.782639:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017745.782645:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2008 00000100:00080000:3.0:1734017745.782649:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017745.782654:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017745.782661:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017745.782675:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332854144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734017745.782680:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017745.782700:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017745.782701:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017745/real 1734017745] req@ffff8800a73cc380 x1818247332853632/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017745.782705:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017745.782707:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017745.782713:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2008 00000100:00080000:3.0:1734017745.782719:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017745.782725:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017745.782728:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332853632/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017745.782732:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1734017745.782739:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017745.782742:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017745.782747:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332853632:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017745.782747:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332854272/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017745.782754:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332853760:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017745.782760:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017745.782765:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017745.782768:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1734017745.782771:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017745.782773:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2008 00000100:00100000:0.0:1734017745.782778:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017745/real 1734017745] req@ffff8800a73cce00 x1818247332853760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017745.782778:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017745.782783:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017745.782789:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332853760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017745.782790:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1734017745.782797:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017745.782800:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017745.782803:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332853760:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734017745.782805:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332854400/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017745.782809:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332853888:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017745.782823:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017745.782829:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017745/real 1734017745] req@ffff8800a73ced80 x1818247332853888/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017745.782839:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332853888/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017745.782847:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017745.782849:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017745.782852:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332853888:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017745.782879:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332854016:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017745.782892:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017745.782898:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017745/real 1734017745] req@ffff8800a73cd880 x1818247332854016/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017745.782908:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332854016/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017745.782915:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017745.782917:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017745.782920:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332854016:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017745.782926:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332854144:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017745.782937:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017745.782943:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017745/real 1734017745] req@ffff8800a73cf480 x1818247332854144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017745.782953:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332854144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017745.782960:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017745.782962:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017745.782965:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332854144:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017745.782970:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332854272:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017745.782985:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017745.782991:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017745/real 1734017745] req@ffff8800a73cca80 x1818247332854272/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017745.783002:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332854272/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017745.783009:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017745.783011:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017745.783014:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332854272:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017745.783019:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332854400:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017745.783030:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017745.783036:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017745/real 1734017745] req@ffff8800a73cd500 x1818247332854400/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017745.783046:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332854400/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017745.783053:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017745.783055:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017745.783058:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332854400:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017746.196507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017747.198487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017748.200582:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017749.202554:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017750.204512:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017750.790363:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017750.790372:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017750.790377:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017750.790388:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2013 00000100:00080000:3.0:1734017750.790398:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017750.790406:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017750.790442:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332854528/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017750.790478:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017750.790483:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017750.790485:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017750.790489:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2013 00000100:00080000:3.0:1734017750.790492:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017750.790497:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017750.790506:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017750.790518:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332854656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017750.790528:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017750.790531:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017750.790533:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017750.790537:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2013 00000100:00080000:3.0:1734017750.790540:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017750.790543:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017750.790548:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017750.790556:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332854784/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017750.790572:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017750.790575:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017750.790577:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017750.790581:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2013 00000100:00080000:3.0:1734017750.790583:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017750.790586:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017750.790590:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017750.790598:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332854912/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017750.790607:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017750.790611:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1734017750.790612:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332854528:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017750.790612:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017750.790616:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2013 00000100:00080000:3.0:1734017750.790619:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017750.790622:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017750.790627:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017750.790635:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332855040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017750.790643:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017750.790646:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017750.790647:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017750.790651:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2013 00000100:00080000:3.0:1734017750.790655:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017750.790658:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017750.790663:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017750.790670:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332855168/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017750.790678:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017750.790681:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017750.790682:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017750.790686:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2013 00000100:00080000:3.0:1734017750.790699:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017750.790702:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734017750.790704:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017750.790707:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017750.790715:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332855296/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017750.790732:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017750/real 1734017750] req@ffff8800a73cea00 x1818247332854528/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017750.790764:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332854528/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017750.790776:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017750.790780:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017750.790786:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332854528:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017750.790794:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332854656:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017750.790813:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017750.790820:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017750/real 1734017750] req@ffff8800a73cd500 x1818247332854656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017750.790832:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332854656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017750.790839:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017750.790842:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017750.790845:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332854656:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017750.790852:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332854784:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017750.790867:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017750.790873:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017750/real 1734017750] req@ffff8800a73cca80 x1818247332854784/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017750.790883:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332854784/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017750.790890:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017750.790894:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017750.790897:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332854784:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017750.790928:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332854912:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017750.790942:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017750.790948:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017750/real 1734017750] req@ffff8800a73cf480 x1818247332854912/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017750.790959:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332854912/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017750.790967:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017750.790969:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017750.790972:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332854912:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017750.790978:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332855040:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017750.790991:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017750.790997:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017750/real 1734017750] req@ffff8800a73cd880 x1818247332855040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017750.791007:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332855040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017750.791015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017750.791017:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017750.791021:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332855040:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017750.791027:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332855168:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017750.791043:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017750.791049:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017750/real 1734017750] req@ffff8800a73ced80 x1818247332855168/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017750.791060:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332855168/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017750.791067:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017750.791069:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017750.791072:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332855168:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017750.791078:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332855296:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017750.791091:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017750.791097:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017750/real 1734017750] req@ffff8800a73cce00 x1818247332855296/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017750.791106:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332855296/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017750.791113:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017750.791116:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017750.791119:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332855296:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017751.206494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017752.208494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017753.210467:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017754.212471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017755.214474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017755.798364:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017755.798373:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017755.798378:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017755.798388:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2018 00000100:00080000:3.0:1734017755.798397:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017755.798404:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017755.798438:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332855424/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017755.798480:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017755.798484:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017755.798486:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017755.798490:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2018 00000100:00080000:3.0:1734017755.798494:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017755.798498:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017755.798506:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017755.798517:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332855552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017755.798526:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017755.798530:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017755.798531:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017755.798535:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2018 00000100:00080000:3.0:1734017755.798538:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017755.798541:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017755.798546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017755.798554:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332855680/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017755.798562:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017755.798565:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017755.798566:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017755.798570:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2018 00000100:00080000:3.0:1734017755.798572:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017755.798576:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017755.798581:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017755.798588:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332855424:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017755.798589:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332855808/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017755.798598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017755.798602:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017755.798603:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017755.798607:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2018 00000100:00080000:3.0:1734017755.798609:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017755.798612:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017755.798616:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017755.798624:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332855936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017755.798633:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017755.798635:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017755.798637:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017755.798641:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2018 00000100:00080000:3.0:1734017755.798644:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017755.798647:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017755.798651:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017755.798654:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017755.798659:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332856064/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017755.798668:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017755.798672:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017755.798673:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017755.798676:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017755/real 1734017755] req@ffff8800a73cc380 x1818247332855424/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017755.798677:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2018 00000100:00080000:3.0:1734017755.798680:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017755.798683:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017755.798687:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017755.798696:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332856192/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017755.798703:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332855424/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017755.798714:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017755.798717:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017755.798722:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332855424:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017755.798729:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332855552:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017755.798746:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017755.798753:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017755/real 1734017755] req@ffff8800a73cf100 x1818247332855552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017755.798764:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332855552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017755.798772:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017755.798774:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017755.798777:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332855552:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017755.798783:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332855680:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017755.798797:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017755.798803:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017755/real 1734017755] req@ffff8800a73cdf80 x1818247332855680/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017755.798813:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332855680/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017755.798820:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017755.798823:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017755.798826:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332855680:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017755.798851:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332855808:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017755.798863:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017755.798869:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017755/real 1734017755] req@ffff8800a73cf800 x1818247332855808/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017755.798880:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332855808/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017755.798887:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017755.798890:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017755.798892:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332855808:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017755.798898:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332855936:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017755.798909:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017755.798915:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017755/real 1734017755] req@ffff8800a73cfb80 x1818247332855936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017755.798925:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332855936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017755.798932:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017755.798934:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017755.798937:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332855936:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017755.798944:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332856064:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017755.798959:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017755.798965:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017755/real 1734017755] req@ffff8800a73cdc00 x1818247332856064/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017755.798975:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332856064/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017755.798983:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017755.798985:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017755.798988:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332856064:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017755.798993:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332856192:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017755.799005:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017755.799011:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017755/real 1734017755] req@ffff8800a73ce680 x1818247332856192/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017755.799020:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332856192/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017755.799027:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017755.799029:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017755.799032:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332856192:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017756.216457:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017757.218475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017758.220514:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017759.222593:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017760.224498:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017760.806357:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017760.806367:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017760.806371:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017760.806381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2023 00000100:00080000:3.0:1734017760.806391:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017760.806399:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017760.806436:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332856320/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017760.806472:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017760.806477:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017760.806479:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017760.806483:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2023 00000100:00080000:3.0:1734017760.806486:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017760.806490:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017760.806501:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017760.806514:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332856448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017760.806525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017760.806528:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017760.806530:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017760.806534:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2023 00000100:00080000:3.0:1734017760.806536:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017760.806540:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017760.806544:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017760.806553:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332856576/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017760.806562:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017760.806565:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017760.806566:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017760.806570:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2023 00000100:00080000:3.0:1734017760.806572:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017760.806575:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017760.806580:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017760.806587:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332856704/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017760.806596:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017760.806600:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017760.806602:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017760.806606:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2023 00000100:00080000:3.0:1734017760.806609:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017760.806612:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017760.806613:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332856320:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017760.806616:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017760.806625:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332856832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017760.806633:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017760.806636:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017760.806637:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017760.806641:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2023 00000100:00080000:3.0:1734017760.806644:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017760.806648:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017760.806652:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017760.806659:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332856960/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017760.806667:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017760.806670:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017760.806672:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017760.806675:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2023 00000100:00080000:3.0:1734017760.806678:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017760.806681:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017760.806686:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017760.806693:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332857088/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734017760.806707:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017760.806736:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017760/real 1734017760] req@ffff8800a73cc700 x1818247332856320/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017760.806767:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332856320/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017760.806779:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017760.806783:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017760.806788:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332856320:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017760.806796:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332856448:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017760.806815:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017760.806822:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017760/real 1734017760] req@ffff8800a73ce680 x1818247332856448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017760.806834:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332856448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017760.806842:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017760.806845:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017760.806848:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332856448:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017760.806854:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332856576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017760.806869:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017760.806875:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017760/real 1734017760] req@ffff8800a73cdc00 x1818247332856576/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017760.806885:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332856576/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017760.806893:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017760.806897:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017760.806900:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332856576:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017760.806931:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332856704:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017760.806944:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017760.806951:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017760/real 1734017760] req@ffff8800a73cfb80 x1818247332856704/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017760.806961:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332856704/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017760.806969:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017760.806971:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017760.806974:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332856704:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017760.806980:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332856832:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017760.806993:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017760.806999:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017760/real 1734017760] req@ffff8800a73cf800 x1818247332856832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017760.807009:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332856832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017760.807018:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017760.807020:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017760.807023:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332856832:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017760.807029:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332856960:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017760.807045:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017760.807051:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017760/real 1734017760] req@ffff8800a73cdf80 x1818247332856960/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017760.807109:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332856960/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017760.807119:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017760.807121:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017760.807125:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332856960:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017760.807131:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332857088:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017760.807149:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017760.807157:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017760/real 1734017760] req@ffff8800a73cf100 x1818247332857088/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017760.807167:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332857088/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017760.807175:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017760.807177:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017760.807180:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332857088:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017761.226480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017762.228472:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017763.230461:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017764.232492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017765.234460:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017765.814377:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017765.814387:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017765.814391:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017765.814402:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2028 00000100:00080000:3.0:1734017765.814413:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017765.814420:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017765.814455:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332857216/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017765.814492:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017765.814497:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017765.814499:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017765.814503:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2028 00000100:00080000:3.0:1734017765.814507:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017765.814511:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017765.814520:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017765.814531:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332857344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017765.814541:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017765.814544:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017765.814546:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017765.814550:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2028 00000100:00080000:3.0:1734017765.814555:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017765.814559:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017765.814564:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017765.814572:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332857472/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017765.814581:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017765.814584:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017765.814585:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017765.814589:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2028 00000100:00080000:3.0:1734017765.814591:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017765.814595:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017765.814599:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017765.814607:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332857600/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017765.814616:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017765.814619:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017765.814621:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017765.814625:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2028 00000100:00080000:3.0:1734017765.814627:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017765.814631:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017765.814632:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332857216:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017765.814635:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017765.814643:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332857728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017765.814658:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017765.814661:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017765.814663:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017765.814667:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2028 00000100:00080000:3.0:1734017765.814669:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017765.814673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017765.814677:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017765.814685:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332857856/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017765.814693:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017765.814696:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017765.814697:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017765.814701:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2028 00000100:00080000:3.0:1734017765.814704:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017765.814707:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017765.814711:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017765.814720:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332857984/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734017765.814725:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017765.814753:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017765/real 1734017765] req@ffff8800a73cc380 x1818247332857216/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017765.814783:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332857216/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017765.814796:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017765.814799:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017765.814804:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332857216:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017765.814812:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332857344:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017765.814831:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017765.814837:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017765/real 1734017765] req@ffff8800a73cce00 x1818247332857344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017765.814848:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332857344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017765.814856:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017765.814859:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017765.814862:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332857344:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017765.814869:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332857472:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017765.814884:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017765.814890:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017765/real 1734017765] req@ffff8800a73ced80 x1818247332857472/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017765.814900:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332857472/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017765.814908:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017765.814911:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017765.814915:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332857472:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017765.814947:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332857600:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017765.814960:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017765.814966:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017765/real 1734017765] req@ffff8800a73cd880 x1818247332857600/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017765.815021:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332857600/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017765.815031:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017765.815033:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017765.815037:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332857600:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017765.815043:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332857728:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017765.815061:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017765.815068:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017765/real 1734017765] req@ffff8800a73cf480 x1818247332857728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017765.815079:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332857728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017765.815087:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017765.815090:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017765.815093:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332857728:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017765.815098:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332857856:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017765.815115:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017765.815121:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017765/real 1734017765] req@ffff8800a73cca80 x1818247332857856/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017765.815132:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332857856/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017765.815139:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017765.815142:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017765.815145:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332857856:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017765.815150:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332857984:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017765.815163:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017765.815169:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017765/real 1734017765] req@ffff8800a73cd500 x1818247332857984/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017765.815179:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332857984/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017765.815186:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017765.815189:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017765.815192:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332857984:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017766.236500:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017767.238518:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017768.240594:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017769.242503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017770.244524:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017770.822366:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017770.822376:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017770.822381:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017770.822393:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2033 00000100:00080000:3.0:1734017770.822403:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017770.822411:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017770.822447:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332858112/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017770.822480:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017770.822485:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017770.822486:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017770.822491:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2033 00000100:00080000:3.0:1734017770.822494:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017770.822498:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017770.822509:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017770.822522:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332858240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017770.822532:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017770.822535:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017770.822537:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017770.822541:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2033 00000100:00080000:3.0:1734017770.822544:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017770.822547:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017770.822552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017770.822561:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332858368/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017770.822577:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017770.822580:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017770.822582:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017770.822585:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332858112:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017770.822585:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2033 00000100:00080000:3.0:1734017770.822588:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017770.822591:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017770.822596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017770.822603:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332858496/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017770.822611:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017770.822615:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017770.822616:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017770.822619:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2033 00000100:00080000:3.0:1734017770.822622:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017770.822625:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017770.822629:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017770.822637:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332858624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017770.822645:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017770.822648:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017770.822649:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1734017770.822653:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017770.822654:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2033 00000100:00080000:3.0:1734017770.822657:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017770.822661:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017770.822665:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017770.822672:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332858752/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017770.822675:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017770/real 1734017770] req@ffff8800a73cea00 x1818247332858112/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017770.822680:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017770.822683:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017770.822684:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017770.822688:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2033 00000100:00080000:3.0:1734017770.822690:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017770.822694:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017770.822698:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017770.822703:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332858112/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017770.822706:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332858880/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017770.822714:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017770.822717:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017770.822722:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332858112:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017770.822729:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332858240:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017770.822747:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017770.822753:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017770/real 1734017770] req@ffff8800a73cd500 x1818247332858240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017770.822765:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332858240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017770.822773:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017770.822775:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017770.822778:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332858240:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017770.822801:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332858368:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017770.822814:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017770.822820:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017770/real 1734017770] req@ffff8800a73cca80 x1818247332858368/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017770.822830:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332858368/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017770.822839:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017770.822842:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017770.822846:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332858368:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017770.822852:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332858496:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017770.822864:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017770.822870:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017770/real 1734017770] req@ffff8800a73cf480 x1818247332858496/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017770.822879:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332858496/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017770.822887:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017770.822889:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017770.822937:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332858496:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017770.822944:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332858624:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017770.822960:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017770.822967:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017770/real 1734017770] req@ffff8800a73cd880 x1818247332858624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017770.822977:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332858624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017770.822985:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017770.822987:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017770.822990:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332858624:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017770.822995:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332858752:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017770.823012:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017770.823018:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017770/real 1734017770] req@ffff8800a73ced80 x1818247332858752/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017770.823029:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332858752/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017770.823037:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017770.823039:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017770.823042:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332858752:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017770.823047:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332858880:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017770.823059:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017770.823064:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017770/real 1734017770] req@ffff8800a73cce00 x1818247332858880/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017770.823074:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332858880/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017770.823081:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017770.823083:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017770.823086:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332858880:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017771.246479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017772.248453:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017773.250468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017774.252494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017775.254454:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017775.830367:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017775.830377:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017775.830381:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017775.830392:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2038 00000100:00080000:3.0:1734017775.830406:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017775.830414:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017775.830450:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332859008/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017775.830493:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017775.830499:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017775.830502:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017775.830508:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2038 00000100:00080000:3.0:1734017775.830516:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017775.830522:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017775.830535:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017775.830553:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332859136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017775.830562:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332859008:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017775.830569:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017775.830573:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017775.830575:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017775.830580:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2038 00000100:00080000:3.0:1734017775.830584:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017775.830589:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017775.830596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017775.830608:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332859264/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017775.830620:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017775.830624:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017775.830627:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017775.830632:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2038 00000800:00000100:0.0:1734017775.830633:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017775.830635:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017775.830641:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017775.830648:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017775.830658:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017775/real 1734017775] req@ffff8800a73cc380 x1818247332859008/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017775.830660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332859392/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017775.830674:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017775.830679:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017775.830681:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017775.830686:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2038 00000100:00080000:3.0:1734017775.830690:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017775.830694:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332859008/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017775.830695:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017775.830702:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1734017775.830709:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017775.830713:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017775.830715:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332859520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017775.830720:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332859008:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017775.830729:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017775.830733:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017775.830735:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017775.830740:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2038 00000100:00080000:3.0:1734017775.830744:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017775.830747:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332859136:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017775.830749:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017775.830756:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017775.830768:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332859648/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734017775.830773:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017775.830782:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017775.830784:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017775/real 1734017775] req@ffff8800a73cf100 x1818247332859136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017775.830788:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017775.830791:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017775.830796:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2038 00000100:00080000:3.0:1734017775.830800:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017775.830801:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332859136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017775.830839:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017775.830848:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1734017775.830865:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1734017775.830865:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332859776/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017775.830869:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017775.830875:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332859136:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017775.830885:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332859264:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017775.830916:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017775.830929:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017775/real 1734017775] req@ffff8800a73cdf80 x1818247332859264/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017775.830948:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332859264/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017775.830962:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017775.830965:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017775.830969:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332859264:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017775.830976:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332859392:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017775.830997:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017775.831004:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017775/real 1734017775] req@ffff8800a73cf800 x1818247332859392/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017775.831016:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332859392/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017775.831024:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017775.831026:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017775.831029:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332859392:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017775.831034:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332859520:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017775.831046:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017775.831052:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017775/real 1734017775] req@ffff8800a73cfb80 x1818247332859520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017775.831062:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332859520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017775.831070:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017775.831072:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017775.831076:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332859520:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017775.831094:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332859648:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017775.831111:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017775.831117:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017775/real 1734017775] req@ffff8800a73cdc00 x1818247332859648/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017775.831128:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332859648/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017775.831136:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017775.831138:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017775.831140:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332859648:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017775.831145:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332859776:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017775.831157:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017775.831163:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017775/real 1734017775] req@ffff8800a73ce680 x1818247332859776/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017775.831172:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332859776/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017775.831179:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017775.831182:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017775.831185:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332859776:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017776.256448:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017777.258480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017778.260510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017779.262459:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017780.264460:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017780.838376:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017780.838385:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017780.838389:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017780.838399:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2043 00000100:00080000:3.0:1734017780.838409:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017780.838416:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017780.838452:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332859904/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017780.838487:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017780.838493:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017780.838496:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017780.838503:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2043 00000100:00080000:3.0:1734017780.838508:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017780.838514:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017780.838528:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017780.838548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332860032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017780.838555:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332859904:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017780.838565:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017780.838569:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017780.838572:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017780.838577:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2043 00000100:00080000:3.0:1734017780.838581:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017780.838585:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017780.838592:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017780.838605:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332860160/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017780.838618:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017780.838622:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1734017780.838625:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017780.838625:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017780.838630:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2043 00000100:00080000:3.0:1734017780.838633:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017780.838638:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017780.838644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017780.838649:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017780/real 1734017780] req@ffff8800a73cc700 x1818247332859904/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017780.838657:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332860288/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017780.838671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017780.838676:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017780.838679:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017780.838684:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2043 00000100:00100000:0.0:1734017780.838685:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332859904/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017780.838688:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017780.838693:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:0.0:1734017780.838700:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1734017780.838700:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1734017780.838705:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017780.838711:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332859904:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017780.838712:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332860416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017780.838764:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017780.838768:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017780.838771:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017780.838777:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2043 00000100:00080000:3.0:1734017780.838782:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017780.838787:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017780.838794:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332860032:192.168.201.152@tcp:38:kworker.0 00000008:00100000:3.0:1734017780.838795:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017780.838809:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332860544/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017780.838824:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1734017780.838826:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017780.838829:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017780.838831:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017780.838837:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2043 00000100:00100000:0.0:1734017780.838839:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017780/real 1734017780] req@ffff8800a73ce680 x1818247332860032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017780.838841:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017780.838847:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017780.838854:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017780.838863:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332860032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017780.838866:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332860672/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017780.838876:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017780.838881:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017780.838887:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332860032:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017780.838896:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332860160:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017780.838923:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017780.838936:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017780/real 1734017780] req@ffff8800a73cdc00 x1818247332860160/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017780.838953:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332860160/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017780.838965:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017780.838968:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017780.838972:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332860160:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017780.838979:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332860288:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017780.838998:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017780.839005:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017780/real 1734017780] req@ffff8800a73cfb80 x1818247332860288/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017780.839016:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332860288/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017780.839024:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017780.839026:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017780.839030:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332860288:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017780.839035:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332860416:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017780.839047:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017780.839053:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017780/real 1734017780] req@ffff8800a73cf800 x1818247332860416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017780.839063:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332860416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017780.839070:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017780.839073:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017780.839076:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332860416:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017780.839094:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332860544:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017780.839110:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017780.839117:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017780/real 1734017780] req@ffff8800a73cdf80 x1818247332860544/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017780.839127:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332860544/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017780.839135:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017780.839137:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017780.839140:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332860544:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017780.839144:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332860672:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017780.839156:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017780.839162:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017780/real 1734017780] req@ffff8800a73cf100 x1818247332860672/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017780.839171:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332860672/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017780.839178:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017780.839180:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017780.839183:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332860672:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017781.266477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017782.268479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017783.270488:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017784.272507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017785.274469:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017785.846333:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017785.846343:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017785.846348:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017785.846358:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2048 00000100:00080000:3.0:1734017785.846368:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017785.846375:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017785.846411:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332860800/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017785.846444:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017785.846449:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017785.846451:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017785.846455:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2048 00000100:00080000:3.0:1734017785.846458:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017785.846462:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017785.846471:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017785.846482:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332860928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017785.846492:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017785.846495:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017785.846497:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017785.846500:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2048 00000100:00080000:3.0:1734017785.846505:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017785.846508:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017785.846513:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017785.846521:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332861056/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017785.846529:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017785.846532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017785.846534:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017785.846537:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2048 00000100:00080000:3.0:1734017785.846539:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017785.846543:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017785.846547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017785.846549:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332860800:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017785.846555:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332861184/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017785.846563:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017785.846567:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017785.846568:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017785.846572:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2048 00000100:00080000:3.0:1734017785.846575:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017785.846578:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017785.846582:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017785.846589:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332861312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017785.846604:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017785.846607:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017785.846608:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017785.846612:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2048 00000800:00000100:0.0:1734017785.846613:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017785.846615:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017785.846618:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017785.846622:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017785.846629:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332861440/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017785.846661:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017785/real 1734017785] req@ffff8800a73cc380 x1818247332860800/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017785.846671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017785.846674:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017785.846676:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017785.846680:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2048 00000100:00080000:3.0:1734017785.846683:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017785.846686:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017785.846690:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332860800/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017785.846691:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017785.846700:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332861568/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017785.846701:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017785.846704:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017785.846709:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332860800:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017785.846717:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332860928:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017785.846737:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017785.846744:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017785/real 1734017785] req@ffff8800a73cce00 x1818247332860928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017785.846755:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332860928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017785.846763:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017785.846766:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017785.846769:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332860928:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017785.846775:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332861056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017785.846788:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017785.846794:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017785/real 1734017785] req@ffff8800a73ced80 x1818247332861056/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017785.846804:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332861056/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017785.846812:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017785.846815:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017785.846818:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332861056:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017785.846843:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332861184:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017785.846856:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017785.846862:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017785/real 1734017785] req@ffff8800a73cd880 x1818247332861184/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017785.846872:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332861184/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017785.846880:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017785.846882:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017785.846885:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332861184:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017785.846890:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332861312:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017785.846902:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017785.846908:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017785/real 1734017785] req@ffff8800a73cf480 x1818247332861312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017785.846917:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332861312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017785.846925:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017785.846927:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017785.846930:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332861312:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017785.846935:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332861440:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017785.846949:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017785.846955:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017785/real 1734017785] req@ffff8800a73cca80 x1818247332861440/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017785.846966:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332861440/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017785.846973:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017785.846975:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017785.846977:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332861440:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017785.846983:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332861568:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017785.846995:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017785.847000:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017785/real 1734017785] req@ffff8800a73cd500 x1818247332861568/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017785.847010:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332861568/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017785.847017:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017785.847019:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017785.847022:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332861568:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017786.276491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017787.278461:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017788.280460:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017789.282478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017790.284487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017790.854350:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017790.854360:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017790.854366:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017790.854376:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2053 00000100:00080000:3.0:1734017790.854385:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017790.854393:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017790.854429:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332861696/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017790.854463:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017790.854467:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017790.854469:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017790.854474:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2053 00000100:00080000:3.0:1734017790.854477:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017790.854481:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017790.854490:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017790.854503:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332861824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017790.854513:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017790.854516:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017790.854518:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017790.854522:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2053 00000100:00080000:3.0:1734017790.854525:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017790.854528:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017790.854533:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017790.854542:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332861952/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017790.854596:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017790.854600:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017790.854601:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017790.854605:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2053 00000100:00080000:3.0:1734017790.854608:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017790.854612:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332861696:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017790.854612:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017790.854617:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017790.854626:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332862080/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017790.854635:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017790.854638:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017790.854640:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017790.854643:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2053 00000100:00080000:3.0:1734017790.854646:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017790.854649:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017790.854653:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017790.854661:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332862208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017790.854669:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017790.854672:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017790.854674:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1734017790.854678:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017790.854678:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2053 00000100:00080000:3.0:1734017790.854681:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017790.854685:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017790.854689:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017790.854696:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332862336/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017790.854699:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017790/real 1734017790] req@ffff8800a73cea00 x1818247332861696/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017790.854704:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017790.854707:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017790.854709:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017790.854712:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2053 00000100:00080000:3.0:1734017790.854715:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017790.854718:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017790.854722:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017790.854727:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332861696/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017790.854730:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332862464/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017790.854737:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017790.854740:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017790.854745:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332861696:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017790.854752:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332861824:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017790.854769:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017790.854776:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017790/real 1734017790] req@ffff8800a73cd500 x1818247332861824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017790.854787:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332861824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017790.854795:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017790.854797:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017790.854801:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332861824:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017790.854807:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332861952:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017790.854819:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017790.854825:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017790/real 1734017790] req@ffff8800a73cca80 x1818247332861952/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017790.854835:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332861952/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017790.854842:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017790.854845:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017790.854849:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332861952:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017790.854872:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332862080:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017790.854885:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017790.854891:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017790/real 1734017790] req@ffff8800a73cf480 x1818247332862080/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017790.854902:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332862080/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017790.854909:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017790.854911:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017790.854914:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332862080:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017790.854920:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332862208:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017790.854931:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017790.854937:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017790/real 1734017790] req@ffff8800a73cd880 x1818247332862208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017790.854947:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332862208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017790.854954:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017790.854956:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017790.854960:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332862208:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017790.854966:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332862336:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017790.854981:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017790.854987:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017790/real 1734017790] req@ffff8800a73ced80 x1818247332862336/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017790.854997:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332862336/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017790.855004:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017790.855006:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017790.855009:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332862336:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017790.855014:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332862464:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017790.855026:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017790.855032:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017790/real 1734017790] req@ffff8800a73cce00 x1818247332862464/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017790.855042:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332862464/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017790.855050:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017790.855052:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017790.855055:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332862464:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017791.286487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017792.288470:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017793.290465:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017794.292472:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017795.294536:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017795.862351:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017795.862360:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017795.862364:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017795.862375:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2058 00000100:00080000:3.0:1734017795.862386:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017795.862394:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017795.862430:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332862592/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017795.862498:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017795.862503:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017795.862505:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017795.862510:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2058 00000100:00080000:3.0:1734017795.862513:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017795.862517:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017795.862527:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017795.862539:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332862720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017795.862549:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017795.862552:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017795.862554:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017795.862557:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2058 00000100:00080000:3.0:1734017795.862560:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017795.862563:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017795.862568:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017795.862576:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332862848/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017795.862577:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332862592:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017795.862585:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017795.862588:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017795.862590:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017795.862593:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2058 00000100:00080000:3.0:1734017795.862596:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017795.862600:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017795.862604:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017795.862612:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332862976/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017795.862620:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017795.862623:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017795.862624:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017795.862627:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2058 00000100:00080000:3.0:1734017795.862640:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017795.862644:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017795.862648:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017795.862656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332863104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017795.862664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017795.862667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017795.862668:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017795.862672:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332862720:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017795.862672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2058 00000100:00080000:3.0:1734017795.862675:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017795.862678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017795.862682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017795.862689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332863232/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017795.862697:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017795.862701:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332862848:192.168.201.152@tcp:8:kworker.0 00000100:00080000:3.0:1734017795.862701:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017795.862703:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017795.862706:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2058 00000100:00080000:3.0:1734017795.862709:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017795.862712:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017795.862716:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017795.862718:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332862976:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734017795.862726:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332863360/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017795.862734:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332863104:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017795.862749:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332863232:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017795.862765:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332863360:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017796.296535:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017797.298504:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017798.300513:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1734017798.868543:0:1957:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:1.0:1734017798.868590:0:1957:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1734017798.868609:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017798.868651:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017798.868658:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017798.868665:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017798.868672:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017798.868678:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017798.868685:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:0.0:1734017798.868792:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017795/real 1734017798] req@ffff8800a73ce680 x1818247332863360/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017798.868843:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332863360/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017798.868863:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017798.868870:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017798.868878:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332863360:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017798.868893:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017795/real 1734017798] req@ffff8800a73cdf80 x1818247332862848/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017798.868916:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332862848/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017798.868929:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017798.868934:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017798.868940:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332862848:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017798.868952:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017795/real 1734017798] req@ffff8800a73cf800 x1818247332862976/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017798.868971:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332862976/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017798.868985:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017798.868990:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017798.868995:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332862976:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017798.869007:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017795/real 1734017798] req@ffff8800a73cfb80 x1818247332863104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017798.869027:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332863104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017798.869040:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017798.869045:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017798.869050:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332863104:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017798.869062:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017795/real 1734017798] req@ffff8800a73cdc00 x1818247332863232/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017798.869082:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332863232/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017798.869094:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017798.869099:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017798.869105:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332863232:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017798.869117:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017795/real 1734017798] req@ffff8800a73cc380 x1818247332862592/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017798.869137:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332862592/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017798.869160:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017798.869165:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017798.869170:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332862592:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017798.869182:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017795/real 1734017798] req@ffff8800a73cf100 x1818247332862720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017798.869201:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332862720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017798.869296:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017798.869301:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017798.869308:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332862720:192.168.201.152@tcp:38:kworker.0 00010000:00010000:2.0:1734017799.302513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017800.304534:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017800.870365:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017800.870375:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017800.870416:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017800.870427:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2063 00000100:00080000:3.0:1734017800.870437:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017800.870445:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017800.870482:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332863488/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017800.870514:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017800.870519:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017800.870521:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017800.870525:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2063 00000100:00080000:3.0:1734017800.870528:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017800.870532:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017800.870543:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017800.870556:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332863616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017800.870566:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017800.870570:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017800.870572:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017800.870575:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2063 00000100:00080000:3.0:1734017800.870578:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017800.870581:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017800.870586:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017800.870594:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332863744/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017800.870603:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017800.870606:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017800.870607:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017800.870611:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2063 00000100:00080000:3.0:1734017800.870613:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017800.870617:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017800.870621:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017800.870622:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332863488:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017800.870629:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332863872/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017800.870637:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017800.870641:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017800.870643:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017800.870646:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2063 00000100:00080000:3.0:1734017800.870649:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017800.870652:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017800.870656:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017800.870664:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332864000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017800.870672:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017800.870675:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017800.870676:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017800.870681:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2063 00000100:00080000:3.0:1734017800.870683:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734017800.870686:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017800.870687:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017800.870691:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017800.870698:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332864128/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017800.870706:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017800.870708:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017800/real 1734017800] req@ffff8800a73cc700 x1818247332863488/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017800.870709:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017800.870711:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017800.870714:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2063 00000100:00080000:3.0:1734017800.870717:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017800.870720:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017800.870724:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017800.870727:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332863488/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017800.870732:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332864256/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017800.870737:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017800.870741:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017800.870745:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332863488:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017800.870752:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332863616:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017800.870769:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017800.870777:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017800/real 1734017800] req@ffff8800a73cf100 x1818247332863616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017800.870788:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332863616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017800.870796:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017800.870798:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017800.870802:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332863616:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017800.870807:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332863744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017800.870820:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017800.870826:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017800/real 1734017800] req@ffff8800a73cc380 x1818247332863744/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017800.870836:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332863744/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017800.870844:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017800.870846:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017800.870849:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332863744:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017800.870873:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332863872:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017800.870886:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017800.870891:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017800/real 1734017800] req@ffff8800a73cdc00 x1818247332863872/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017800.870902:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332863872/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017800.870916:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017800.870919:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017800.870922:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332863872:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017800.870927:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332864000:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017800.870939:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017800.870945:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017800/real 1734017800] req@ffff8800a73cfb80 x1818247332864000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017800.870954:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332864000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017800.870962:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017800.870964:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017800.870968:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332864000:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017800.870973:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332864128:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017800.870985:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017800.870990:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017800/real 1734017800] req@ffff8800a73cf800 x1818247332864128/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017800.871000:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332864128/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017800.871007:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017800.871010:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017800.871013:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332864128:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017800.871019:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332864256:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017800.871031:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017800.871037:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017800/real 1734017800] req@ffff8800a73cdf80 x1818247332864256/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017800.871047:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332864256/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017800.871054:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017800.871056:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017800.871059:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332864256:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017801.306520:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017802.308498:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017803.310426:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017804.312503:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017805.314522:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017805.878389:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017805.878398:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017805.878402:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017805.878412:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2068 00000100:00080000:3.0:1734017805.878422:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017805.878430:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017805.878466:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332864384/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017805.878499:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017805.878503:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017805.878505:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017805.878509:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2068 00000100:00080000:3.0:1734017805.878512:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017805.878516:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017805.878526:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017805.878536:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332864512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017805.878546:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017805.878549:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017805.878551:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017805.878554:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2068 00000100:00080000:3.0:1734017805.878559:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017805.878562:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017805.878568:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017805.878576:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332864640/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017805.878584:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017805.878587:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017805.878589:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017805.878592:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2068 00000100:00080000:3.0:1734017805.878595:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017805.878598:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017805.878602:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017805.878610:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332864384:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017805.878610:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332864768/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017805.878619:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017805.878622:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017805.878624:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017805.878627:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2068 00000100:00080000:3.0:1734017805.878629:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017805.878633:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017805.878637:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017805.878644:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332864896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017805.878659:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017805.878662:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017805.878664:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017805.878667:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2068 00000100:00080000:3.0:1734017805.878670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734017805.878674:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017805.878674:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017805.878678:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017805.878685:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332865024/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017805.878693:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017805.878696:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017805/real 1734017805] req@ffff8800a73ce680 x1818247332864384/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017805.878696:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017805.878698:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017805.878701:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2068 00000100:00080000:3.0:1734017805.878704:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017805.878707:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017805.878711:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017805.878716:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332864384/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017805.878720:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332865152/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017805.878726:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017805.878729:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017805.878734:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332864384:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017805.878741:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332864512:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017805.878758:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017805.878765:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017805/real 1734017805] req@ffff8800a73cce00 x1818247332864512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017805.878784:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332864512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017805.878792:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017805.878794:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017805.878797:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332864512:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017805.878803:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332864640:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017805.878816:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017805.878822:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017805/real 1734017805] req@ffff8800a73ced80 x1818247332864640/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017805.878832:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332864640/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017805.878839:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017805.878841:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017805.878844:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332864640:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017805.878870:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332864768:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017805.878883:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017805.878889:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017805/real 1734017805] req@ffff8800a73cd880 x1818247332864768/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017805.878899:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332864768/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017805.878907:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017805.878910:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017805.878913:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332864768:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017805.878919:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332864896:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017805.878930:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017805.878936:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017805/real 1734017805] req@ffff8800a73cf480 x1818247332864896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017805.878946:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332864896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017805.878954:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017805.878956:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017805.878959:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332864896:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017805.878965:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332865024:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017805.878976:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017805.878982:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017805/real 1734017805] req@ffff8800a73cca80 x1818247332865024/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017805.878992:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332865024/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017805.878999:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017805.879001:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017805.879004:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332865024:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017805.879010:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332865152:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017805.879025:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017805.879030:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017805/real 1734017805] req@ffff8800a73cd500 x1818247332865152/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017805.879041:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332865152/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017805.879048:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017805.879050:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017805.879053:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332865152:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017806.316501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017807.318565:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017808.320481:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017809.322472:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017810.324467:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017810.886371:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017810.886380:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017810.886385:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017810.886395:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2073 00000100:00080000:3.0:1734017810.886404:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017810.886412:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017810.886448:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332865280/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017810.886482:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017810.886486:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017810.886488:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017810.886492:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2073 00000100:00080000:3.0:1734017810.886496:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017810.886499:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017810.886509:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017810.886522:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332865408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017810.886532:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017810.886535:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017810.886537:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017810.886540:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2073 00000100:00080000:3.0:1734017810.886543:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017810.886547:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017810.886552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017810.886560:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332865536/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017810.886576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017810.886579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017810.886581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017810.886584:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2073 00000100:00080000:3.0:1734017810.886586:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017810.886590:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017810.886593:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332865280:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017810.886594:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017810.886601:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332865664/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017810.886610:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017810.886613:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017810.886615:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017810.886618:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2073 00000100:00080000:3.0:1734017810.886621:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017810.886624:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017810.886628:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017810.886635:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332865792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017810.886643:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017810.886646:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017810.886648:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017810.886651:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2073 00000100:00080000:3.0:1734017810.886655:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017810.886658:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734017810.886659:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017810.886662:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017810.886670:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332865920/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017810.886677:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017810.886680:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734017810.886681:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017810/real 1734017810] req@ffff8800a73cea00 x1818247332865280/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017810.886682:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017810.886685:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2073 00000100:00080000:3.0:1734017810.886688:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017810.886691:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017810.886695:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017810.886700:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332865280/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017810.886702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332866048/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017810.886711:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017810.886714:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017810.886719:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332865280:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017810.886726:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332865408:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017810.886743:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017810.886749:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017810/real 1734017810] req@ffff8800a73cd500 x1818247332865408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017810.886760:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332865408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017810.886769:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017810.886772:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017810.886775:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332865408:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017810.886781:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332865536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017810.886794:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017810.886800:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017810/real 1734017810] req@ffff8800a73cca80 x1818247332865536/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017810.886810:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332865536/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017810.886817:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017810.886820:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017810.886823:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332865536:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017810.886848:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332865664:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017810.886860:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017810.886866:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017810/real 1734017810] req@ffff8800a73cf480 x1818247332865664/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017810.886877:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332865664/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017810.886884:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017810.886886:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017810.886889:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332865664:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017810.886895:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332865792:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017810.886907:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017810.886913:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017810/real 1734017810] req@ffff8800a73cd880 x1818247332865792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017810.886923:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332865792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017810.886930:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017810.886933:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017810.886936:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332865792:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017810.886941:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332865920:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017810.886952:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017810.886958:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017810/real 1734017810] req@ffff8800a73ced80 x1818247332865920/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017810.886968:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332865920/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017810.886975:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017810.886977:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017810.886980:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332865920:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017810.886985:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332866048:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017810.886998:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017810.887004:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017810/real 1734017810] req@ffff8800a73cce00 x1818247332866048/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017810.887015:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332866048/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017810.887022:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017810.887024:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017810.887027:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332866048:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017811.326496:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017812.328460:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017813.330481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017814.332482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017815.334485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017815.894336:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017815.894345:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017815.894349:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017815.894360:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2078 00000100:00080000:3.0:1734017815.894369:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017815.894377:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017815.894411:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332866176/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017815.894452:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017815.894456:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017815.894458:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017815.894462:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2078 00000100:00080000:3.0:1734017815.894465:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017815.894469:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017815.894479:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017815.894489:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332866304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017815.894499:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017815.894502:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017815.894503:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017815.894507:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2078 00000100:00080000:3.0:1734017815.894510:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017815.894513:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017815.894517:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017815.894525:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332866432/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017815.894533:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017815.894536:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017815.894538:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017815.894541:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2078 00000100:00080000:3.0:1734017815.894544:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017815.894548:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017815.894552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017815.894557:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332866176:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017815.894560:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332866560/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017815.894569:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017815.894572:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017815.894573:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017815.894577:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2078 00000100:00080000:3.0:1734017815.894579:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017815.894582:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017815.894587:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017815.894594:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332866688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017815.894602:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017815.894605:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017815.894607:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017815.894610:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2078 00000100:00080000:3.0:1734017815.894613:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017815.894617:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734017815.894621:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017815.894621:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017815.894628:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332866816/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017815.894636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017815.894640:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734017815.894642:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017815/real 1734017815] req@ffff8800a73ce680 x1818247332866176/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017815.894642:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017815.894645:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2078 00000100:00080000:3.0:1734017815.894648:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017815.894651:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017815.894655:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017815.894662:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332866176/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017815.894664:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332866944/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017815.894673:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017815.894676:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017815.894681:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332866176:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017815.894688:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332866304:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017815.894706:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017815.894712:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017815/real 1734017815] req@ffff8800a73cdf80 x1818247332866304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017815.894724:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332866304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017815.894731:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017815.894735:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017815.894738:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332866304:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017815.894745:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332866432:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017815.894758:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017815.894764:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017815/real 1734017815] req@ffff8800a73cf800 x1818247332866432/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017815.894774:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332866432/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017815.894782:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017815.894784:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017815.894788:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332866432:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017815.894812:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332866560:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017815.894825:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017815.894831:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017815/real 1734017815] req@ffff8800a73cfb80 x1818247332866560/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017815.894842:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332866560/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017815.894849:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017815.894851:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017815.894854:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332866560:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017815.894860:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332866688:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017815.894875:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017815.894881:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017815/real 1734017815] req@ffff8800a73cdc00 x1818247332866688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017815.894891:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332866688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017815.894899:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017815.894901:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017815.894904:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332866688:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017815.894909:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332866816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017815.894921:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017815.894927:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017815/real 1734017815] req@ffff8800a73cc380 x1818247332866816/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017815.894936:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332866816/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017815.894944:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017815.894946:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017815.894949:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332866816:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017815.894954:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332866944:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017815.894966:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017815.894972:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017815/real 1734017815] req@ffff8800a73cf100 x1818247332866944/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017815.894988:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332866944/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017815.894996:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017815.894998:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017815.895001:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332866944:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017816.336457:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017817.338428:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017818.340450:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017819.342386:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017820.344481:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017820.902360:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017820.902369:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017820.902373:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017820.902383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2083 00000100:00080000:3.0:1734017820.902392:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017820.902400:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017820.902437:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332867072/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017820.902470:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017820.902474:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017820.902476:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017820.902481:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2083 00000100:00080000:3.0:1734017820.902484:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017820.902488:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017820.902499:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017820.902511:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332867200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017820.902521:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017820.902525:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017820.902526:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017820.902530:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2083 00000100:00080000:3.0:1734017820.902532:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017820.902536:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017820.902540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017820.902549:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332867328/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017820.902557:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017820.902560:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017820.902562:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017820.902565:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2083 00000100:00080000:3.0:1734017820.902567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017820.902571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017820.902575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017820.902578:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332867072:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017820.902582:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332867456/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017820.902591:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017820.902594:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017820.902596:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017820.902599:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2083 00000100:00080000:3.0:1734017820.902602:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017820.902605:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017820.902609:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017820.902617:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332867584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017820.902625:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017820.902628:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017820.902629:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017820.902633:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2083 00000100:00080000:3.0:1734017820.902636:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017820.902639:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017820.902643:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017820.902644:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017820.902651:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332867712/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017820.902658:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017820.902662:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017820.902663:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017820.902666:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2083 00000100:00100000:0.0:1734017820.902667:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017820/real 1734017820] req@ffff8800a73cc700 x1818247332867072/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017820.902669:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017820.902672:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017820.902677:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017820.902684:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332867840/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017820.902685:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332867072/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017820.902696:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017820.902699:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017820.902703:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332867072:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017820.902711:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332867200:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017820.902728:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017820.902735:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017820/real 1734017820] req@ffff8800a73cf100 x1818247332867200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017820.902746:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332867200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017820.902754:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017820.902758:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017820.902761:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332867200:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017820.902767:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332867328:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017820.902781:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017820.902786:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017820/real 1734017820] req@ffff8800a73cc380 x1818247332867328/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017820.902796:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332867328/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017820.902804:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017820.902806:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017820.902810:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332867328:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017820.902834:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332867456:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017820.902846:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017820.902852:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017820/real 1734017820] req@ffff8800a73cdc00 x1818247332867456/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017820.902863:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332867456/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017820.902870:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017820.902873:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017820.902876:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332867456:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017820.902881:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332867584:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017820.902896:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017820.902902:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017820/real 1734017820] req@ffff8800a73cfb80 x1818247332867584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017820.902913:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332867584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017820.902920:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017820.902922:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017820.902925:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332867584:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017820.902931:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332867712:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017820.902943:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017820.902948:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017820/real 1734017820] req@ffff8800a73cf800 x1818247332867712/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017820.902958:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332867712/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017820.902965:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017820.902967:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017820.902971:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332867712:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017820.902977:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332867840:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017820.902988:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017820.902994:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017820/real 1734017820] req@ffff8800a73cdf80 x1818247332867840/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017820.903010:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332867840/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017820.903017:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017820.903019:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017820.903022:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332867840:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017821.346442:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017822.348468:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017823.350467:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017824.352448:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017825.354498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017825.910357:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017825.910366:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017825.910371:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017825.910381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2088 00000100:00080000:3.0:1734017825.910390:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017825.910398:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017825.910433:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332867968/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017825.910467:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017825.910471:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017825.910473:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017825.910478:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2088 00000100:00080000:3.0:1734017825.910481:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017825.910485:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017825.910495:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017825.910506:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332868096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017825.910516:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017825.910520:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017825.910521:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017825.910525:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2088 00000100:00080000:3.0:1734017825.910530:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017825.910533:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017825.910538:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017825.910546:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332868224/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017825.910554:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017825.910557:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017825.910559:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017825.910562:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2088 00000100:00080000:3.0:1734017825.910564:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017825.910568:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017825.910572:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017825.910579:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332867968:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017825.910580:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332868352/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017825.910589:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017825.910592:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017825.910593:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017825.910597:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2088 00000100:00080000:3.0:1734017825.910599:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017825.910602:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017825.910606:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017825.910614:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332868480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017825.910629:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017825.910632:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017825.910634:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017825.910638:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2088 00000100:00080000:3.0:1734017825.910640:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017825.910644:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734017825.910645:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017825.910648:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017825.910656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332868608/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017825.910664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017825.910665:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017825/real 1734017825] req@ffff8800a73ce680 x1818247332867968/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017825.910667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017825.910668:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017825.910671:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2088 00000100:00080000:3.0:1734017825.910674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017825.910677:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017825.910681:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017825.910684:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332867968/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017825.910690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332868736/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017825.910695:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017825.910698:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017825.910703:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332867968:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017825.910711:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332868096:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017825.910728:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017825.910735:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017825/real 1734017825] req@ffff8800a73cce00 x1818247332868096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017825.910746:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332868096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017825.910754:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017825.910757:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017825.910761:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332868096:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017825.910767:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332868224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017825.910781:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017825.910787:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017825/real 1734017825] req@ffff8800a73ced80 x1818247332868224/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017825.910797:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332868224/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017825.910804:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017825.910807:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017825.910810:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332868224:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017825.910835:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332868352:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017825.910847:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017825.910853:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017825/real 1734017825] req@ffff8800a73cd880 x1818247332868352/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017825.910864:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332868352/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017825.910871:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017825.910874:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017825.910877:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332868352:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017825.910882:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332868480:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017825.910903:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017825.910909:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017825/real 1734017825] req@ffff8800a73cf480 x1818247332868480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017825.910919:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332868480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017825.910927:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017825.910929:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017825.910932:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332868480:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017825.910937:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332868608:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017825.910949:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017825.911011:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017825/real 1734017825] req@ffff8800a73cca80 x1818247332868608/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017825.911023:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332868608/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017825.911030:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017825.911032:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017825.911035:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332868608:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017825.911041:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332868736:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017825.911058:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017825.911065:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017825/real 1734017825] req@ffff8800a73cd500 x1818247332868736/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017825.911082:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332868736/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017825.911090:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017825.911092:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017825.911095:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332868736:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017826.356421:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017827.358478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017828.360454:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017829.362495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017830.364462:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017830.918357:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017830.918367:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017830.918372:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017830.918383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2093 00000100:00080000:3.0:1734017830.918393:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017830.918401:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017830.918438:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332868864/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017830.918474:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017830.918480:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017830.918483:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017830.918489:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2093 00000100:00080000:3.0:1734017830.918494:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017830.918501:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017830.918514:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017830.918535:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332868992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017830.918542:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332868864:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017830.918552:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017830.918556:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017830.918558:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017830.918563:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2093 00000100:00080000:3.0:1734017830.918567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017830.918571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017830.918578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017830.918590:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332869120/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734017830.918611:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017830.918612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017830.918617:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017830.918619:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017830.918623:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2093 00000100:00080000:3.0:1734017830.918629:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017830.918634:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017830/real 1734017830] req@ffff8800a73cea00 x1818247332868864/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017830.918634:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017830.918640:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017830.918652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332869248/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017830.918657:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332868864/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017830.918666:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017830.918671:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734017830.918672:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017830.918673:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734017830.918676:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734017830.918679:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2093 00000100:00100000:0.0:1734017830.918682:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332868864:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017830.918683:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017830.918688:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017830.918693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017830.918706:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332869376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017830.918709:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332868992:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017830.918721:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017830.918725:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017830.918728:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017830.918734:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2093 00000800:00000100:0.0:1734017830.918736:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017830.918739:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017830.918745:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017830.918747:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017830/real 1734017830] req@ffff8800a73cd500 x1818247332868992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017830.918752:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017830.918764:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332869504/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017830.918765:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332868992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017830.918778:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017830.918778:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734017830.918783:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734017830.918783:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017830.918786:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017830.918788:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332868992:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017830.918791:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2093 00000100:00080000:3.0:1734017830.918795:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017830.918797:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332869120:192.168.201.152@tcp:8:kworker.0 00000100:00080000:3.0:1734017830.918800:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017830.918806:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017830.918818:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332869632/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734017830.918819:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017830.918829:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017830/real 1734017830] req@ffff8800a73cca80 x1818247332869120/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017830.918846:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332869120/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017830.918858:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017830.918862:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017830.918914:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332869120:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017830.918927:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332869248:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017830.918957:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017830.918968:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017830/real 1734017830] req@ffff8800a73cf480 x1818247332869248/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017830.918980:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332869248/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017830.918989:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017830.918991:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017830.918994:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332869248:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017830.919012:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332869376:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017830.919031:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017830.919037:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017830/real 1734017830] req@ffff8800a73cd880 x1818247332869376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017830.919048:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332869376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017830.919055:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017830.919057:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017830.919060:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332869376:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017830.919065:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332869504:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017830.919077:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017830.919083:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017830/real 1734017830] req@ffff8800a73ced80 x1818247332869504/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017830.919092:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332869504/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017830.919100:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017830.919102:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017830.919105:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332869504:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017830.919110:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332869632:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017830.919122:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017830.919128:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017830/real 1734017830] req@ffff8800a73cce00 x1818247332869632/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017830.919143:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332869632/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017830.919151:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017830.919153:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017830.919156:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332869632:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017831.366484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017832.368467:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017833.370479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017834.372497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017835.374467:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017835.926332:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017835.926342:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017835.926346:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017835.926356:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2098 00000100:00080000:3.0:1734017835.926366:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017835.926374:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017835.926408:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332869760/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017835.926450:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017835.926456:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017835.926459:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017835.926465:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2098 00000100:00080000:3.0:1734017835.926470:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017835.926477:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017835.926489:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017835.926508:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332869888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017835.926519:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332869760:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017835.926522:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017835.926527:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017835.926530:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017835.926534:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2098 00000100:00080000:3.0:1734017835.926538:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017835.926543:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017835.926550:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017835.926561:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332870016/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017835.926574:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017835.926578:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017835.926581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017835.926585:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2098 00000800:00000100:0.0:1734017835.926588:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017835.926589:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017835.926595:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017835.926601:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017835.926611:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017835/real 1734017835] req@ffff8800a73ce680 x1818247332869760/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017835.926613:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332870144/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017835.926627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017835.926632:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017835.926634:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017835.926635:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332869760/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017835.926640:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2098 00000100:00080000:3.0:1734017835.926644:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734017835.926649:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017835.926649:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:0.0:1734017835.926653:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000008:00100000:3.0:1734017835.926655:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734017835.926660:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332869760:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017835.926667:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332870272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017835.926679:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017835.926683:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1734017835.926685:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332869888:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017835.926686:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017835.926691:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2098 00000100:00080000:3.0:1734017835.926695:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017835.926700:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017835.926707:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017835.926711:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017835.926719:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332870400/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017835.926722:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017835/real 1734017835] req@ffff8800a73cdf80 x1818247332869888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017835.926735:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017835.926739:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332869888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017835.926740:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017835.926743:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017835.926748:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2098 00000100:00080000:0.0:1734017835.926752:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017835.926753:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017835.926760:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:0.0:1734017835.926762:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000008:00100000:3.0:1734017835.926766:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017835.926767:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332869888:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017835.926776:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332870016:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734017835.926778:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332870528/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734017835.926837:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017835.926849:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017835/real 1734017835] req@ffff8800a73cf800 x1818247332870016/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017835.926867:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332870016/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017835.926880:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017835.926883:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017835.926889:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332870016:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017835.926898:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332870144:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017835.926924:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017835.926935:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017835/real 1734017835] req@ffff8800a73cfb80 x1818247332870144/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017835.926951:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332870144/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017835.926963:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017835.926967:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017835.926972:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332870144:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017835.926980:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332870272:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017835.927003:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017835.927010:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017835/real 1734017835] req@ffff8800a73cdc00 x1818247332870272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017835.927021:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332870272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017835.927029:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017835.927031:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017835.927034:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332870272:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017835.927053:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332870400:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017835.927065:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017835.927071:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017835/real 1734017835] req@ffff8800a73cc380 x1818247332870400/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017835.927081:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332870400/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017835.927088:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017835.927090:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017835.927094:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332870400:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017835.927099:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332870528:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017835.927110:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017835.927116:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017835/real 1734017835] req@ffff8800a73cf100 x1818247332870528/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017835.927132:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332870528/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017835.927139:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017835.927141:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017835.927144:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332870528:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017836.376443:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017837.378474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017838.380472:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017839.382485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017840.384485:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017840.934373:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017840.934382:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017840.934387:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017840.934398:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2103 00000100:00080000:3.0:1734017840.934407:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017840.934414:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017840.934452:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332870656/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017840.934485:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017840.934489:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017840.934491:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017840.934496:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2103 00000100:00080000:3.0:1734017840.934499:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017840.934503:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017840.934515:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017840.934527:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332870784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017840.934537:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017840.934541:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017840.934542:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017840.934546:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2103 00000100:00080000:3.0:1734017840.934548:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017840.934552:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017840.934557:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017840.934565:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332870912/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017840.934573:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017840.934576:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017840.934578:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017840.934581:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2103 00000100:00080000:3.0:1734017840.934583:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017840.934587:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017840.934591:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017840.934595:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332870656:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017840.934599:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332871040/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017840.934609:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017840.934612:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017840.934614:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017840.934618:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2103 00000100:00080000:3.0:1734017840.934621:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017840.934624:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017840.934628:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017840.934636:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332871168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017840.934644:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017840.934647:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017840.934649:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017840.934653:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2103 00000100:00080000:3.0:1734017840.934656:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017840.934660:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734017840.934661:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017840.934664:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017840.934671:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332871296/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017840.934679:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017840.934682:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734017840.934683:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017840/real 1734017840] req@ffff8800a73cc700 x1818247332870656/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017840.934684:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017840.934687:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2103 00000100:00080000:3.0:1734017840.934690:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017840.934693:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017840.934737:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332870656/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017840.934738:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1734017840.934748:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1734017840.934749:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332871424/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017840.934752:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017840.934757:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332870656:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017840.934764:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332870784:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017840.934785:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017840.934793:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017840/real 1734017840] req@ffff8800a73cf100 x1818247332870784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017840.934805:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332870784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017840.934813:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017840.934817:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017840.934820:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332870784:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017840.934826:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332870912:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017840.934840:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017840.934846:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017840/real 1734017840] req@ffff8800a73cc380 x1818247332870912/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017840.934856:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332870912/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017840.934864:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017840.934866:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017840.934870:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332870912:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017840.934894:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332871040:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017840.934906:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017840.934912:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017840/real 1734017840] req@ffff8800a73cdc00 x1818247332871040/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017840.934922:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332871040/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017840.934930:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017840.934932:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017840.934935:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332871040:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017840.934940:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332871168:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017840.934957:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017840.934963:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017840/real 1734017840] req@ffff8800a73cfb80 x1818247332871168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017840.934973:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332871168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017840.934981:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017840.934983:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017840.934986:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332871168:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017840.934992:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332871296:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017840.935003:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017840.935009:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017840/real 1734017840] req@ffff8800a73cf800 x1818247332871296/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017840.935018:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332871296/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017840.935025:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017840.935027:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017840.935030:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332871296:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017840.935036:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332871424:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017840.935048:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017840.935054:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017840/real 1734017840] req@ffff8800a73cdf80 x1818247332871424/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017840.935071:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332871424/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017840.935079:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017840.935081:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017840.935083:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332871424:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017841.386457:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017842.388497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017843.390462:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017844.392466:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017845.394474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017845.942351:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017845.942361:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017845.942365:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017845.942375:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2108 00000100:00080000:3.0:1734017845.942384:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017845.942392:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017845.942427:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332871552/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017845.942459:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017845.942463:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017845.942465:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017845.942470:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2108 00000100:00080000:3.0:1734017845.942473:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017845.942477:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017845.942486:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017845.942496:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332871680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017845.942506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017845.942509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017845.942511:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017845.942514:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2108 00000100:00080000:3.0:1734017845.942519:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017845.942523:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017845.942532:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017845.942541:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332871808/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017845.942549:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017845.942552:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017845.942554:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017845.942557:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2108 00000100:00080000:3.0:1734017845.942559:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017845.942562:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017845.942567:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017845.942569:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332871552:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017845.942575:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332871936/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017845.942584:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017845.942587:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017845.942588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017845.942592:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2108 00000100:00080000:3.0:1734017845.942594:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017845.942597:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017845.942601:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017845.942609:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332872064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017845.942658:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017845.942661:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017845.942663:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017845.942667:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2108 00000100:00080000:3.0:1734017845.942671:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017845.942674:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734017845.942679:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017845.942679:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017845.942688:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332872192/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017845.942696:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017845.942700:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734017845.942701:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017845/real 1734017845] req@ffff8800a73ce680 x1818247332871552/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017845.942701:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017845.942705:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2108 00000100:00080000:3.0:1734017845.942707:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017845.942710:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017845.942715:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017845.942721:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332871552/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017845.942724:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332872320/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017845.942731:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017845.942735:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017845.942740:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332871552:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017845.942747:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332871680:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017845.942767:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017845.942773:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017845/real 1734017845] req@ffff8800a73cce00 x1818247332871680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017845.942784:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332871680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017845.942792:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017845.942796:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017845.942799:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332871680:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017845.942806:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332871808:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017845.942820:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017845.942825:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017845/real 1734017845] req@ffff8800a73ced80 x1818247332871808/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017845.942835:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332871808/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017845.942843:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017845.942846:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017845.942849:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332871808:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017845.942873:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332871936:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017845.942885:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017845.942891:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017845/real 1734017845] req@ffff8800a73cd880 x1818247332871936/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017845.942901:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332871936/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017845.942909:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017845.942911:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017845.942914:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332871936:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017845.942920:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332872064:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017845.942935:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017845.942940:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017845/real 1734017845] req@ffff8800a73cf480 x1818247332872064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017845.942951:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332872064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017845.942959:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017845.942961:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017845.942964:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332872064:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017845.942969:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332872192:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017845.942981:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017845.942987:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017845/real 1734017845] req@ffff8800a73cca80 x1818247332872192/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017845.942997:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332872192/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017845.943004:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017845.943006:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017845.943009:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332872192:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017845.943014:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332872320:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017845.943026:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017845.943032:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017845/real 1734017845] req@ffff8800a73cd500 x1818247332872320/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017845.943049:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332872320/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017845.943056:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017845.943058:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017845.943061:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332872320:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017846.396475:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017847.398492:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017848.400474:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017849.402463:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017850.404478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017850.950342:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017850.950351:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017850.950356:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017850.950367:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2113 00000100:00080000:3.0:1734017850.950377:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017850.950384:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017850.950420:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332872448/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017850.950452:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017850.950456:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017850.950458:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017850.950463:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2113 00000100:00080000:3.0:1734017850.950466:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017850.950470:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017850.950484:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017850.950497:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332872576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017850.950506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017850.950510:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017850.950511:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017850.950515:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2113 00000100:00080000:3.0:1734017850.950517:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017850.950521:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017850.950559:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017850.950570:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332872704/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017850.950587:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017850.950591:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017850.950592:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017850.950595:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332872448:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017850.950596:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2113 00000100:00080000:3.0:1734017850.950598:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017850.950602:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017850.950606:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017850.950614:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332872832/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017850.950623:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017850.950626:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017850.950627:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017850.950631:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2113 00000100:00080000:3.0:1734017850.950633:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017850.950637:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017850.950641:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017850.950649:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332872960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017850.950657:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017850.950660:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017850.950662:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1734017850.950663:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017850.950666:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2113 00000100:00080000:3.0:1734017850.950670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017850.950673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017850.950677:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017850.950685:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017850/real 1734017850] req@ffff8800a73cea00 x1818247332872448/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017850.950685:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332873088/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017850.950693:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017850.950696:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017850.950698:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017850.950701:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2113 00000100:00080000:3.0:1734017850.950703:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017850.950704:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332872448/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017850.950707:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017850.950711:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1734017850.950714:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017850.950718:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017850.950719:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332873216/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017850.950723:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332872448:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017850.950730:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332872576:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017850.950747:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017850.950754:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017850/real 1734017850] req@ffff8800a73cd500 x1818247332872576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017850.950765:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332872576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017850.950772:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017850.950776:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017850.950779:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332872576:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017850.950803:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332872704:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017850.950817:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017850.950823:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017850/real 1734017850] req@ffff8800a73cca80 x1818247332872704/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017850.950834:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332872704/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017850.950841:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017850.950844:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017850.950847:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332872704:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017850.950852:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332872832:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017850.950864:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017850.950870:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017850/real 1734017850] req@ffff8800a73cf480 x1818247332872832/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017850.950879:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332872832/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017850.950887:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017850.950889:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017850.950892:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332872832:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017850.950898:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332872960:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017850.950918:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017850.950924:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017850/real 1734017850] req@ffff8800a73cd880 x1818247332872960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017850.950934:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332872960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017850.950942:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017850.950944:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017850.950947:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332872960:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017850.950953:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332873088:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017850.950965:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017850.950971:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017850/real 1734017850] req@ffff8800a73ced80 x1818247332873088/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017850.950981:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332873088/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017850.950988:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017850.950991:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017850.950994:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332873088:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017850.950999:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332873216:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017850.951011:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017850.951017:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017850/real 1734017850] req@ffff8800a73cce00 x1818247332873216/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017850.951033:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332873216/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017850.951041:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017850.951043:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017850.951046:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332873216:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017851.406480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017852.408469:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017853.410540:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017854.412461:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017855.414437:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017855.958370:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017855.958380:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017855.958384:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017855.958394:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2118 00000100:00080000:3.0:1734017855.958404:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017855.958412:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017855.958491:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332873344/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017855.958530:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017855.958534:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017855.958536:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017855.958541:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2118 00000100:00080000:3.0:1734017855.958544:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017855.958548:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017855.958558:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017855.958570:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332873472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017855.958580:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017855.958583:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017855.958585:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017855.958588:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2118 00000100:00080000:3.0:1734017855.958590:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017855.958594:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017855.958598:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017855.958607:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332873600/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017855.958615:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017855.958618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017855.958619:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017855.958622:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2118 00000100:00080000:3.0:1734017855.958625:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017855.958629:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017855.958633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017855.958638:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332873344:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017855.958641:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332873728/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017855.958649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017855.958653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017855.958654:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017855.958657:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2118 00000100:00080000:3.0:1734017855.958660:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017855.958663:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017855.958667:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017855.958674:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332873856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017855.958682:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017855.958685:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017855.958687:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017855.958690:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2118 00000100:00080000:3.0:1734017855.958694:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017855.958698:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017855.958702:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017855.958704:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017855.958710:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332873984/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017855.958718:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017855.958722:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017855.958724:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017855.958725:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017855/real 1734017855] req@ffff8800a73ce680 x1818247332873344/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017855.958727:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2118 00000100:00080000:3.0:1734017855.958730:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017855.958733:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017855.958738:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017855.958744:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332873344/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017855.958746:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332874112/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017855.958756:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017855.958759:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017855.958764:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332873344:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017855.958771:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332873472:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017855.958789:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017855.958795:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017855/real 1734017855] req@ffff8800a73cdf80 x1818247332873472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017855.958806:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332873472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017855.958814:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017855.958818:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017855.958821:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332873472:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017855.958828:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332873600:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017855.958841:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017855.958847:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017855/real 1734017855] req@ffff8800a73cf800 x1818247332873600/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017855.958857:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332873600/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017855.958865:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017855.958868:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017855.958871:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332873600:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017855.958896:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332873728:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017855.958908:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017855.958914:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017855/real 1734017855] req@ffff8800a73cfb80 x1818247332873728/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017855.958925:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332873728/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017855.958933:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017855.958935:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017855.958938:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332873728:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017855.958943:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332873856:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017855.958958:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017855.958964:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017855/real 1734017855] req@ffff8800a73cdc00 x1818247332873856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017855.958975:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332873856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017855.958982:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017855.958984:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017855.958987:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332873856:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017855.958993:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332873984:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017855.959004:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017855.959010:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017855/real 1734017855] req@ffff8800a73cc380 x1818247332873984/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017855.959020:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332873984/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017855.959027:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017855.959030:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017855.959032:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332873984:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017855.959038:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332874112:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017855.959050:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017855.959055:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017855/real 1734017855] req@ffff8800a73cf100 x1818247332874112/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017855.959072:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332874112/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017855.959080:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017855.959082:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017855.959085:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332874112:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017856.416522:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017857.418506:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017858.420486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017859.422492:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017860.424467:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017860.966342:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017860.966351:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017860.966394:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017860.966406:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2123 00000100:00080000:3.0:1734017860.966416:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017860.966424:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017860.966460:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332874240/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017860.966492:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017860.966496:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017860.966498:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017860.966503:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2123 00000100:00080000:3.0:1734017860.966506:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017860.966510:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017860.966520:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017860.966534:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332874368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017860.966544:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017860.966547:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017860.966549:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017860.966553:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2123 00000100:00080000:3.0:1734017860.966555:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017860.966558:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017860.966563:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017860.966571:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332874496/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017860.966580:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017860.966583:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017860.966584:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017860.966588:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2123 00000100:00080000:3.0:1734017860.966590:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017860.966593:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017860.966598:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017860.966601:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332874240:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017860.966606:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332874624/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017860.966614:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017860.966618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017860.966620:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017860.966623:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2123 00000100:00080000:3.0:1734017860.966626:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017860.966629:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017860.966633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017860.966641:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332874752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017860.966649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017860.966652:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017860.966653:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017860.966657:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2123 00000100:00080000:3.0:1734017860.966659:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017860.966663:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017860.966667:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017860.966674:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332874880/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017860.966681:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017860.966685:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017860.966686:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017860.966689:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2123 00000100:00080000:3.0:1734017860.966692:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017860.966693:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332874368:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017860.966695:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017860.966699:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017860.966707:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332875008/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017860.966713:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332874496:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017860.966735:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332874624:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017860.966749:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332874752:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017860.966764:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332874880:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017860.966778:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332875008:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017861.426503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017862.428464:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017863.430490:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1734017863.972513:0:1958:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:1.0:1734017863.972558:0:1958:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1734017863.972578:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017863.972612:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017863.972620:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017863.972627:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017863.972634:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017863.972640:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017863.972646:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:2.0:1734017863.972702:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017860/real 1734017863] req@ffff8800a73cdc00 x1818247332874624/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017863.972753:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332874624/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017863.972769:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017863.972774:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017863.972782:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332874624:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017863.972794:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017860/real 1734017863] req@ffff8800a73cfb80 x1818247332874752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017863.972813:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332874752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017863.972825:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017863.972830:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017863.972836:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332874752:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017863.972846:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017860/real 1734017863] req@ffff8800a73cf800 x1818247332874880/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017863.972864:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332874880/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017863.972876:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017863.972881:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017863.972886:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332874880:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017863.972897:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017860/real 1734017863] req@ffff8800a73cdf80 x1818247332875008/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017863.972927:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332875008/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017863.972940:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017863.972944:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017863.972949:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332875008:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017863.972959:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017860/real 1734017863] req@ffff8800a73cc700 x1818247332874240/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017863.972978:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332874240/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017863.972991:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017863.972995:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017863.973000:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332874240:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017863.973011:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017860/real 1734017863] req@ffff8800a73cf100 x1818247332874368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017863.973028:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332874368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017863.973041:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017863.973045:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017863.973050:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332874368:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017863.973060:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017860/real 1734017863] req@ffff8800a73cc380 x1818247332874496/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017863.973087:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332874496/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017863.973100:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017863.973104:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017863.973109:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332874496:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017864.432499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017865.434492:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017865.974365:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017865.974374:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017865.974379:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017865.974389:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2128 00000100:00080000:3.0:1734017865.974399:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017865.974406:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017865.974444:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332875136/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017865.974476:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017865.974480:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017865.974482:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017865.974486:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2128 00000100:00080000:3.0:1734017865.974489:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017865.974493:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017865.974503:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017865.974514:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332875264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017865.974524:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017865.974527:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017865.974529:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017865.974532:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2128 00000100:00080000:3.0:1734017865.974537:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017865.974540:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017865.974545:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017865.974553:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332875392/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017865.974561:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017865.974564:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017865.974566:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017865.974569:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2128 00000100:00080000:3.0:1734017865.974571:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017865.974575:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017865.974579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017865.974586:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332875136:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017865.974587:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332875520/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017865.974596:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017865.974599:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017865.974600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017865.974604:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2128 00000100:00080000:3.0:1734017865.974606:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017865.974609:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017865.974613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017865.974621:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332875648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017865.974636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017865.974639:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017865.974640:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017865.974644:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2128 00000100:00080000:3.0:1734017865.974647:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017865.974650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:2.0:1734017865.974651:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017865.974655:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017865.974662:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332875776/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017865.974670:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734017865.974672:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017865/real 1734017865] req@ffff8800a73ce680 x1818247332875136/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017865.974673:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017865.974674:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017865.974678:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2128 00000100:00080000:3.0:1734017865.974680:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017865.974683:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017865.974687:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017865.974692:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332875136/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017865.974696:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332875904/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017865.974702:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017865.974705:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017865.974710:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332875136:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017865.974717:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332875264:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017865.974735:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017865.974742:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017865/real 1734017865] req@ffff8800a73cce00 x1818247332875264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017865.974753:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332875264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017865.974768:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017865.974770:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017865.974774:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332875264:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017865.974780:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332875392:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017865.974793:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017865.974799:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017865/real 1734017865] req@ffff8800a73ced80 x1818247332875392/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017865.974809:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332875392/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017865.974816:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017865.974819:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017865.974822:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332875392:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017865.974846:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332875520:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017865.974859:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017865.974864:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017865/real 1734017865] req@ffff8800a73cd880 x1818247332875520/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017865.974875:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332875520/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017865.974882:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017865.974885:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017865.974887:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332875520:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017865.974894:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332875648:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017865.974906:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017865.974911:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017865/real 1734017865] req@ffff8800a73cf480 x1818247332875648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017865.974921:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332875648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017865.974929:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017865.974931:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017865.974934:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332875648:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017865.974939:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332875776:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017865.974950:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017865.974956:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017865/real 1734017865] req@ffff8800a73cca80 x1818247332875776/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017865.974966:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332875776/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017865.974974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017865.974976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017865.974979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332875776:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017865.974985:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332875904:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017865.974996:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017865.975002:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017865/real 1734017865] req@ffff8800a73cd500 x1818247332875904/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017865.975012:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332875904/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017865.975020:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017865.975022:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017865.975025:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332875904:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017866.436499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017867.438471:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017868.440478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017869.442477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017870.444501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017870.982368:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017870.982377:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017870.982382:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017870.982394:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2133 00000100:00080000:3.0:1734017870.982404:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017870.982412:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017870.982449:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332876032/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017870.982484:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017870.982488:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017870.982490:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017870.982495:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2133 00000100:00080000:3.0:1734017870.982498:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017870.982502:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017870.982512:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017870.982524:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332876160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017870.982534:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017870.982538:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017870.982539:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017870.982543:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2133 00000100:00080000:3.0:1734017870.982546:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017870.982549:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017870.982554:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017870.982562:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332876288/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017870.982577:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017870.982581:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017870.982582:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017870.982586:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2133 00000100:00080000:3.0:1734017870.982588:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017870.982592:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017870.982596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017870.982597:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332876032:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017870.982603:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332876416/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017870.982612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017870.982616:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017870.982617:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017870.982621:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2133 00000100:00080000:3.0:1734017870.982623:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017870.982626:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017870.982631:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017870.982638:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332876544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017870.982646:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017870.982649:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017870.982651:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017870.982655:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2133 00000100:00080000:3.0:1734017870.982658:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017870.982662:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:2.0:1734017870.982663:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017870.982666:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017870.982673:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332876672/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017870.982681:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017870.982684:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1734017870.982685:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017870/real 1734017870] req@ffff8800a73cea00 x1818247332876032/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017870.982685:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017870.982689:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2133 00000100:00080000:3.0:1734017870.982691:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017870.982694:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017870.982698:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017870.982705:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332876032/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017870.982706:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332876800/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017870.982716:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017870.982720:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017870.982724:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332876032:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017870.982732:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332876160:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017870.982749:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017870.982755:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017870/real 1734017870] req@ffff8800a73cd500 x1818247332876160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017870.982766:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332876160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017870.982775:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017870.982777:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017870.982781:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332876160:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017870.982786:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332876288:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017870.982799:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017870.982805:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017870/real 1734017870] req@ffff8800a73cca80 x1818247332876288/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017870.982815:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332876288/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017870.982822:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017870.982825:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017870.982828:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332876288:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017870.982852:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332876416:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017870.982864:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017870.982870:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017870/real 1734017870] req@ffff8800a73cf480 x1818247332876416/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017870.982880:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332876416/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017870.982888:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017870.982890:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017870.982893:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332876416:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017870.982899:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332876544:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017870.982911:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017870.982917:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017870/real 1734017870] req@ffff8800a73cd880 x1818247332876544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017870.982927:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332876544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017870.982935:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017870.982937:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017870.982940:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332876544:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017870.982946:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332876672:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017870.982957:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017870.982962:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017870/real 1734017870] req@ffff8800a73ced80 x1818247332876672/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017870.982972:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332876672/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017870.982979:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017870.982981:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017870.982984:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332876672:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017870.982990:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332876800:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017870.983002:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017870.983008:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017870/real 1734017870] req@ffff8800a73cce00 x1818247332876800/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017870.983019:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332876800/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017870.983026:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017870.983028:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017870.983031:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332876800:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017871.446505:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017872.448491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017873.450483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017874.452463:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017875.454492:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017875.990358:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017875.990367:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017875.990372:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017875.990382:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2138 00000100:00080000:3.0:1734017875.990393:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017875.990401:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017875.990434:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332876928/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017875.990474:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017875.990479:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017875.990481:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017875.990485:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2138 00000100:00080000:3.0:1734017875.990488:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017875.990492:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017875.990502:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017875.990512:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332877056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017875.990522:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017875.990525:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017875.990527:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017875.990530:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2138 00000100:00080000:3.0:1734017875.990533:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017875.990536:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017875.990541:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017875.990549:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332877184/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017875.990557:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017875.990560:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017875.990562:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017875.990565:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2138 00000100:00080000:3.0:1734017875.990567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017875.990571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017875.990576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017875.990581:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332876928:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017875.990583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332877312/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017875.990592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017875.990595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017875.990597:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017875.990600:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2138 00000100:00080000:3.0:1734017875.990603:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017875.990606:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017875.990610:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017875.990617:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332877440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017875.990625:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017875.990628:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017875.990630:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017875.990634:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2138 00000100:00080000:3.0:1734017875.990636:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017875.990640:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017875.990644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734017875.990647:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017875.990651:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332877568/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017875.990660:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017875.990663:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017875.990665:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734017875.990668:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017875/real 1734017875] req@ffff8800a73ce680 x1818247332876928/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017875.990668:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2138 00000100:00080000:3.0:1734017875.990671:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017875.990675:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017875.990679:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017875.990687:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332876928/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017875.990688:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332877696/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017875.990698:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017875.990701:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017875.990706:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332876928:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017875.990713:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332877056:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017875.990730:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017875.990737:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017875/real 1734017875] req@ffff8800a73cc380 x1818247332877056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017875.990748:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332877056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017875.990757:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017875.990760:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017875.990763:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332877056:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017875.990769:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332877184:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017875.990782:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017875.990788:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017875/real 1734017875] req@ffff8800a73cf100 x1818247332877184/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017875.990798:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332877184/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017875.990805:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017875.990808:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017875.990811:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332877184:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017875.990835:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332877312:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017875.990847:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017875.990853:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017875/real 1734017875] req@ffff8800a73cc700 x1818247332877312/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017875.990863:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332877312/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017875.990871:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017875.990873:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017875.990876:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332877312:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017875.990883:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332877440:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017875.990895:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017875.990901:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017875/real 1734017875] req@ffff8800a73cdf80 x1818247332877440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017875.990911:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332877440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017875.990918:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017875.990921:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017875.990924:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332877440:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017875.990929:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332877568:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017875.990941:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017875.990947:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017875/real 1734017875] req@ffff8800a73cf800 x1818247332877568/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017875.990957:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332877568/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017875.990964:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017875.990966:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017875.990969:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332877568:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017875.990974:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332877696:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017875.990985:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017875.990991:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017875/real 1734017875] req@ffff8800a73cfb80 x1818247332877696/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017875.991001:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332877696/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017875.991008:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017875.991010:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017875.991013:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332877696:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017876.456514:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017877.458498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017878.460508:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017879.462366:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017880.464463:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017880.998307:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017880.998319:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017880.998325:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017880.998339:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2143 00000100:00080000:3.0:1734017880.998351:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017880.998360:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017880.998410:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332877824/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017880.998450:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017880.998454:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017880.998456:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017880.998462:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2143 00000100:00080000:3.0:1734017880.998465:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017880.998469:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017880.998482:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017880.998496:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332877952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017880.998507:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017880.998511:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017880.998513:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017880.998517:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2143 00000100:00080000:3.0:1734017880.998520:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017880.998524:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017880.998529:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017880.998538:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332878080/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017880.998548:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017880.998552:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017880.998553:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017880.998556:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2143 00000100:00080000:3.0:1734017880.998559:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017880.998563:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:2.0:1734017880.998566:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332877824:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017880.998568:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017880.998577:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332878208/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017880.998585:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017880.998589:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017880.998591:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017880.998595:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2143 00000100:00080000:3.0:1734017880.998598:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017880.998602:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017880.998606:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017880.998615:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332878336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017880.998624:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017880.998627:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017880.998629:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017880.998633:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2143 00000100:00080000:3.0:1734017880.998636:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734017880.998639:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017880.998640:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017880.998644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017880.998652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332878464/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017880.998661:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734017880.998663:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017880/real 1734017880] req@ffff8800a73cdc00 x1818247332877824/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017880.998664:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017880.998665:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017880.998669:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2143 00000100:00080000:3.0:1734017880.998673:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017880.998676:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017880.998681:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017880.998683:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332877824/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017880.998690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332878592/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017880.998695:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017880.998698:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017880.998704:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332877824:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017880.998713:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332877952:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017880.998731:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017880.998738:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017880/real 1734017880] req@ffff8800a73cfb80 x1818247332877952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017880.998749:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332877952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017880.998758:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017880.998762:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017880.998765:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332877952:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017880.998789:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332878080:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017880.998804:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017880.998810:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017880/real 1734017880] req@ffff8800a73cf800 x1818247332878080/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017880.998821:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332878080/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017880.998829:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017880.998831:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017880.998835:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332878080:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017880.998840:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332878208:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017880.998852:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017880.998858:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017880/real 1734017880] req@ffff8800a73cdf80 x1818247332878208/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017880.998868:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332878208/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017880.998875:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017880.998878:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017880.998881:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332878208:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017880.998887:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332878336:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017880.998901:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017880.998907:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017880/real 1734017880] req@ffff8800a73cc700 x1818247332878336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017880.998917:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332878336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017880.998925:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017880.998928:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017880.998930:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332878336:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017880.998937:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332878464:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017880.998950:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017880.998955:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017880/real 1734017880] req@ffff8800a73cf100 x1818247332878464/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017880.998965:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332878464/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017880.998972:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017880.998975:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017880.998978:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332878464:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017880.998984:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332878592:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017880.998996:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017880.999002:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017880/real 1734017880] req@ffff8800a73cc380 x1818247332878592/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017880.999013:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332878592/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017880.999057:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017880.999060:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017880.999063:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332878592:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017881.466460:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017882.468488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017883.470504:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017884.472450:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017885.474485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017886.006355:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017886.006366:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017886.006372:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017886.006386:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2148 00000100:00080000:3.0:1734017886.006397:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017886.006406:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017886.006455:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332878720/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017886.006492:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017886.006496:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017886.006498:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017886.006504:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2148 00000100:00080000:3.0:1734017886.006507:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017886.006511:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017886.006522:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017886.006534:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332878848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017886.006544:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017886.006547:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017886.006549:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017886.006553:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2148 00000100:00080000:3.0:1734017886.006558:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017886.006562:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017886.006568:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017886.006577:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332878976/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017886.006585:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017886.006588:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017886.006590:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017886.006593:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2148 00000100:00080000:3.0:1734017886.006596:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017886.006599:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017886.006604:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017886.006612:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332878720:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017886.006613:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332879104/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017886.006623:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017886.006626:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017886.006628:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017886.006632:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2148 00000100:00080000:3.0:1734017886.006637:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017886.006640:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017886.006645:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017886.006653:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332879232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017886.006668:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017886.006671:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017886.006673:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017886.006678:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2148 00000100:00080000:3.0:1734017886.006681:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734017886.006685:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017886.006685:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017886.006689:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017886.006697:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332879360/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017886.006706:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734017886.006708:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017886/real 1734017886] req@ffff8800a73ce680 x1818247332878720/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017886.006709:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017886.006710:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017886.006714:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2148 00000100:00080000:3.0:1734017886.006717:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017886.006721:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017886.006725:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017886.006728:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332878720/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017886.006736:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332879488/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017886.006741:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017886.006744:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017886.006749:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332878720:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017886.006757:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332878848:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017886.006775:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017886.006782:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017886/real 1734017886] req@ffff8800a73cce00 x1818247332878848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017886.006793:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332878848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017886.006802:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017886.006805:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017886.006809:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332878848:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017886.006815:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332878976:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017886.006829:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017886.006835:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017886/real 1734017886] req@ffff8800a73ced80 x1818247332878976/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017886.006846:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332878976/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017886.006853:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017886.006856:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017886.006859:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332878976:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017886.006886:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332879104:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017886.006898:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017886.006904:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017886/real 1734017886] req@ffff8800a73cd880 x1818247332879104/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017886.006915:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332879104/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017886.006922:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017886.006925:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017886.006927:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332879104:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017886.006973:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332879232:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017886.006989:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017886.006997:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017886/real 1734017886] req@ffff8800a73cf480 x1818247332879232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017886.007008:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332879232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017886.007017:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017886.007019:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017886.007022:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332879232:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017886.007028:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332879360:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017886.007040:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017886.007046:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017886/real 1734017886] req@ffff8800a73cca80 x1818247332879360/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017886.007056:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332879360/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017886.007064:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017886.007066:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017886.007069:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332879360:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017886.007075:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332879488:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017886.007087:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017886.007092:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017886/real 1734017886] req@ffff8800a73cd500 x1818247332879488/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017886.007103:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332879488/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017886.007111:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017886.007113:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017886.007116:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332879488:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017886.476480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017887.478487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017888.480516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017889.482491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017890.484474:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017891.014335:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017891.014345:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017891.014352:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017891.014367:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2153 00000100:00080000:3.0:1734017891.014380:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017891.014389:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017891.014438:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332879616/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017891.014476:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017891.014481:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017891.014483:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017891.014488:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2153 00000100:00080000:3.0:1734017891.014492:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017891.014496:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017891.014509:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017891.014523:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332879744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017891.014533:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017891.014537:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017891.014538:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017891.014542:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2153 00000100:00080000:3.0:1734017891.014545:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017891.014549:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017891.014555:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017891.014564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332879872/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017891.014582:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017891.014586:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017891.014587:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017891.014591:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2153 00000100:00080000:3.0:1734017891.014594:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017891.014597:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:2.0:1734017891.014601:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332879616:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017891.014603:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017891.014611:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332880000/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017891.014619:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017891.014623:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017891.014624:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017891.014628:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2153 00000100:00080000:3.0:1734017891.014631:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017891.014635:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017891.014640:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017891.014648:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332880128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017891.014657:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017891.014660:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017891.014662:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017891.014666:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2153 00000100:00080000:3.0:1734017891.014670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734017891.014673:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017891.014674:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017891.014680:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017891.014688:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332880256/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017891.014696:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734017891.014698:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017891/real 1734017891] req@ffff8800a73cea00 x1818247332879616/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017891.014700:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017891.014701:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017891.014705:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2153 00000100:00080000:3.0:1734017891.014708:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017891.014711:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017891.014715:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017891.014717:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332879616/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017891.014725:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332880384/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017891.014729:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017891.014733:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017891.014738:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332879616:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017891.014747:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332879744:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017891.014765:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017891.014772:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017891/real 1734017891] req@ffff8800a73cd500 x1818247332879744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017891.014783:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332879744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017891.014792:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017891.014796:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017891.014799:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332879744:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017891.014824:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332879872:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017891.014838:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017891.014844:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017891/real 1734017891] req@ffff8800a73cca80 x1818247332879872/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017891.014888:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332879872/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017891.014896:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017891.014899:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017891.014902:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332879872:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017891.014908:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332880000:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017891.014923:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017891.014930:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017891/real 1734017891] req@ffff8800a73cf480 x1818247332880000/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017891.014940:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332880000/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017891.014948:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017891.014950:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017891.014953:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332880000:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017891.014960:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332880128:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017891.014973:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017891.014979:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017891/real 1734017891] req@ffff8800a73cd880 x1818247332880128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017891.014990:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332880128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017891.014997:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017891.015000:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017891.015003:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332880128:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017891.015009:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332880256:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017891.015021:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017891.015027:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017891/real 1734017891] req@ffff8800a73ced80 x1818247332880256/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017891.015037:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332880256/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017891.015044:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017891.015046:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017891.015049:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332880256:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017891.015055:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332880384:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017891.015067:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017891.015073:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017891/real 1734017891] req@ffff8800a73cce00 x1818247332880384/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017891.015084:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332880384/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017891.015092:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017891.015094:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017891.015097:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332880384:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017891.486491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017892.488476:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017893.490497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017894.492471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017895.494456:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017896.022340:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017896.022349:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017896.022354:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017896.022364:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2158 00000100:00080000:3.0:1734017896.022374:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017896.022381:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017896.022417:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332880512/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017896.022458:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017896.022462:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017896.022464:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017896.022469:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2158 00000100:00080000:3.0:1734017896.022472:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017896.022476:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017896.022486:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017896.022496:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332880640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017896.022506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017896.022509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017896.022511:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017896.022514:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2158 00000100:00080000:3.0:1734017896.022517:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017896.022520:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017896.022525:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017896.022533:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332880768/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017896.022541:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017896.022544:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017896.022545:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017896.022549:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2158 00000100:00080000:3.0:1734017896.022551:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017896.022555:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017896.022559:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017896.022562:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332880512:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017896.022567:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332880896/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017896.022576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017896.022579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017896.022580:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017896.022584:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2158 00000100:00080000:3.0:1734017896.022586:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017896.022589:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017896.022594:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017896.022602:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332881024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017896.022610:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017896.022613:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017896.022614:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017896.022618:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2158 00000100:00080000:3.0:1734017896.022621:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017896.022624:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:2.0:1734017896.022625:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017896.022628:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017896.022636:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332881152/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017896.022644:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734017896.022646:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017896/real 1734017896] req@ffff8800a73ce680 x1818247332880512/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017896.022648:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017896.022649:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017896.022653:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2158 00000100:00080000:3.0:1734017896.022656:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017896.022659:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017896.022663:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017896.022666:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332880512/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017896.022673:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332881280/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017896.022677:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017896.022680:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017896.022685:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332880512:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017896.022692:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332880640:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017896.022709:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017896.022716:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017896/real 1734017896] req@ffff8800a73cc380 x1818247332880640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017896.022727:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332880640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017896.022736:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017896.022738:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017896.022742:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332880640:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017896.022747:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332880768:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017896.022795:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017896.022803:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017896/real 1734017896] req@ffff8800a73cf100 x1818247332880768/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017896.022814:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332880768/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017896.022822:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017896.022824:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017896.022828:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332880768:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017896.022853:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332880896:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017896.022869:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017896.022875:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017896/real 1734017896] req@ffff8800a73cc700 x1818247332880896/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017896.022886:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332880896/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017896.022893:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017896.022895:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017896.022898:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332880896:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017896.022904:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332881024:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017896.022917:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017896.022922:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017896/real 1734017896] req@ffff8800a73cdf80 x1818247332881024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017896.022933:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332881024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017896.022940:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017896.022943:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017896.022946:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332881024:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017896.022951:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332881152:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017896.022962:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017896.022968:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017896/real 1734017896] req@ffff8800a73cf800 x1818247332881152/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017896.022978:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332881152/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017896.022985:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017896.022987:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017896.022990:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332881152:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017896.022995:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332881280:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017896.023007:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017896.023013:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017896/real 1734017896] req@ffff8800a73cfb80 x1818247332881280/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017896.023023:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332881280/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017896.023031:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017896.023033:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017896.023036:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332881280:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017896.496486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017897.498504:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017898.500459:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017899.502450:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017900.504465:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017901.030355:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017901.030364:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017901.030368:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017901.030379:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2163 00000100:00080000:3.0:1734017901.030389:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017901.030396:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017901.030434:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332881408/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017901.030466:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017901.030471:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017901.030473:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017901.030477:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2163 00000100:00080000:3.0:1734017901.030481:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017901.030485:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017901.030500:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017901.030513:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332881536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017901.030523:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017901.030527:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017901.030529:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017901.030532:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2163 00000100:00080000:3.0:1734017901.030535:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017901.030538:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017901.030543:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017901.030552:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332881664/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017901.030561:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017901.030564:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017901.030566:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734017901.030568:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332881408:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017901.030569:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2163 00000100:00080000:3.0:1734017901.030572:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017901.030575:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017901.030579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017901.030587:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332881792/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017901.030595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017901.030599:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017901.030601:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017901.030604:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2163 00000100:00080000:3.0:1734017901.030624:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017901.030627:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017901.030632:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1734017901.030633:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017901.030641:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332881920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017901.030649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017901.030653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017901.030654:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734017901.030655:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017901/real 1734017901] req@ffff8800a73cdc00 x1818247332881408/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017901.030658:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2163 00000100:00080000:3.0:1734017901.030661:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017901.030664:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017901.030669:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017901.030702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332882048/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734017901.030703:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332881408/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017901.030711:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017901.030714:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017901.030714:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017901.030716:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734017901.030717:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734017901.030719:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2163 00000100:00080000:3.0:1734017901.030721:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734017901.030723:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332881408:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017901.030725:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017901.030730:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017901.030731:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332881536:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734017901.030739:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332882176/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734017901.030751:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017901.030758:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017901/real 1734017901] req@ffff8800a73cfb80 x1818247332881536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017901.030770:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332881536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017901.030779:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017901.030782:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017901.030785:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332881536:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017901.030808:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332881664:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017901.030822:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017901.030828:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017901/real 1734017901] req@ffff8800a73cf800 x1818247332881664/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017901.030838:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332881664/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017901.030846:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017901.030848:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017901.030851:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332881664:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017901.030856:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332881792:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017901.030867:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017901.030873:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017901/real 1734017901] req@ffff8800a73cdf80 x1818247332881792/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017901.030882:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332881792/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017901.030890:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017901.030892:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017901.030895:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332881792:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017901.030901:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332881920:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017901.030912:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017901.030918:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017901/real 1734017901] req@ffff8800a73cc700 x1818247332881920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017901.030928:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332881920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017901.030936:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017901.030938:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017901.030941:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332881920:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017901.030947:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332882048:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017901.030958:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017901.030964:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017901/real 1734017901] req@ffff8800a73cf100 x1818247332882048/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017901.030974:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332882048/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017901.030981:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017901.030983:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017901.030986:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332882048:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017901.030992:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332882176:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017901.031003:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017901.031009:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017901/real 1734017901] req@ffff8800a73cc380 x1818247332882176/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017901.031020:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332882176/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017901.031028:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017901.031030:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017901.031033:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332882176:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017901.506477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017902.508412:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017903.510470:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017904.512477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017905.514449:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017906.038326:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017906.038335:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017906.038339:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017906.038349:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2168 00000100:00080000:3.0:1734017906.038359:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017906.038366:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017906.038401:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332882304/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017906.038433:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017906.038438:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017906.038439:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017906.038444:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2168 00000100:00080000:3.0:1734017906.038447:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017906.038451:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017906.038460:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017906.038471:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332882432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017906.038481:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017906.038484:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017906.038486:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017906.038489:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2168 00000100:00080000:3.0:1734017906.038494:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017906.038497:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017906.038502:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017906.038510:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332882560/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017906.038519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017906.038522:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017906.038523:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017906.038526:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2168 00000100:00080000:3.0:1734017906.038529:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017906.038532:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017906.038536:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017906.038539:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332882304:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017906.038544:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332882688/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017906.038553:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017906.038556:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017906.038557:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017906.038561:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2168 00000100:00080000:3.0:1734017906.038563:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017906.038566:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017906.038570:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017906.038578:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332882816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017906.038626:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017906.038630:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017906.038631:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017906.038636:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2168 00000100:00080000:3.0:1734017906.038639:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017906.038642:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:2.0:1734017906.038643:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017906.038647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017906.038656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332882944/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017906.038664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734017906.038666:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017906/real 1734017906] req@ffff8800a73ce680 x1818247332882304/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017906.038667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017906.038669:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017906.038672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2168 00000100:00080000:3.0:1734017906.038675:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017906.038678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017906.038682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017906.038686:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332882304/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017906.038691:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332883072/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017906.038697:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017906.038700:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017906.038705:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332882304:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017906.038712:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332882432:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017906.038732:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017906.038738:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017906/real 1734017906] req@ffff8800a73cce00 x1818247332882432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017906.038749:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332882432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017906.038758:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017906.038761:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017906.038764:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332882432:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017906.038770:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332882560:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017906.038783:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017906.038789:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017906/real 1734017906] req@ffff8800a73ced80 x1818247332882560/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017906.038799:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332882560/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017906.038806:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017906.038809:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017906.038812:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332882560:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017906.038837:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332882688:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017906.038849:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017906.038855:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017906/real 1734017906] req@ffff8800a73cd880 x1818247332882688/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017906.038865:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332882688/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017906.038873:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017906.038875:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017906.038878:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332882688:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017906.038884:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332882816:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017906.038896:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017906.038902:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017906/real 1734017906] req@ffff8800a73cf480 x1818247332882816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017906.038913:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332882816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017906.038920:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017906.038923:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017906.038925:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332882816:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017906.038931:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332882944:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017906.038942:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017906.038948:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017906/real 1734017906] req@ffff8800a73cca80 x1818247332882944/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017906.038957:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332882944/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017906.038964:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017906.038967:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017906.038969:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332882944:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017906.038975:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332883072:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017906.038987:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017906.038993:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017906/real 1734017906] req@ffff8800a73cd500 x1818247332883072/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017906.039004:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332883072/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017906.039011:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017906.039013:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017906.039016:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332883072:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017906.516423:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017907.518462:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017908.520470:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017909.522489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017910.524478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017911.046355:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017911.046364:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017911.046369:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017911.046379:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2173 00000100:00080000:3.0:1734017911.046390:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017911.046398:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017911.046435:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332883200/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017911.046469:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017911.046473:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017911.046475:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017911.046479:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2173 00000100:00080000:3.0:1734017911.046483:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017911.046486:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017911.046496:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017911.046562:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332883328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017911.046574:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017911.046577:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017911.046579:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017911.046582:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2173 00000100:00080000:3.0:1734017911.046585:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734017911.046589:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332883200:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017911.046589:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017911.046595:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017911.046604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332883456/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017911.046620:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017911.046623:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017911.046625:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017911.046628:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2173 00000100:00080000:3.0:1734017911.046646:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017911.046650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:2.0:1734017911.046654:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017911.046655:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017911.046663:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332883584/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017911.046672:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017911.046675:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1734017911.046676:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017911/real 1734017911] req@ffff8800a73cea00 x1818247332883200/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017911.046677:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017911.046680:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2173 00000100:00080000:3.0:1734017911.046684:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017911.046687:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017911.046691:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734017911.046696:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332883200/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017911.046699:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332883712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017911.046707:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734017911.046708:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017911.046710:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734017911.046711:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734017911.046711:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734017911.046716:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332883200:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017911.046716:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2173 00000100:00080000:3.0:1734017911.046719:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017911.046723:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:2.0:1734017911.046724:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332883328:192.168.201.152@tcp:38:kworker.0 00000008:00100000:3.0:1734017911.046727:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017911.046735:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332883840/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734017911.046741:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017911.046742:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017911.046746:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1734017911.046747:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017911/real 1734017911] req@ffff8800a73cd500 x1818247332883328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017911.046747:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017911.046750:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2173 00000100:00080000:3.0:1734017911.046753:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017911.046756:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:2.0:1734017911.046759:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332883328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017911.046761:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1734017911.046768:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1734017911.046769:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332883968/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017911.046771:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017911.046775:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332883328:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017911.046797:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332883456:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017911.046811:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017911.046817:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017911/real 1734017911] req@ffff8800a73cca80 x1818247332883456/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017911.046828:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332883456/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017911.046835:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017911.046838:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017911.046841:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332883456:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017911.046846:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332883584:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017911.046857:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017911.046863:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017911/real 1734017911] req@ffff8800a73cf480 x1818247332883584/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017911.046873:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332883584/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017911.046880:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017911.046882:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017911.046885:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332883584:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017911.046892:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332883712:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017911.046904:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017911.046910:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017911/real 1734017911] req@ffff8800a73cd880 x1818247332883712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017911.046921:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332883712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017911.046928:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017911.046930:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017911.046933:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332883712:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017911.046939:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332883840:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017911.046951:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017911.046956:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017911/real 1734017911] req@ffff8800a73ced80 x1818247332883840/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017911.046966:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332883840/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017911.046973:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017911.046975:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017911.046979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332883840:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017911.046984:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332883968:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017911.046996:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017911.047001:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017911/real 1734017911] req@ffff8800a73cce00 x1818247332883968/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017911.047012:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332883968/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017911.047020:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017911.047022:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017911.047024:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332883968:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017911.526486:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017912.528534:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017913.530517:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017914.532530:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017915.534510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017916.054349:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017916.054358:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017916.054362:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017916.054373:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2178 00000100:00080000:3.0:1734017916.054384:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017916.054391:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017916.054456:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332884096/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017916.054494:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017916.054499:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017916.054501:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017916.054505:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2178 00000100:00080000:3.0:1734017916.054509:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017916.054513:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017916.054523:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017916.054535:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332884224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017916.054544:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017916.054547:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017916.054549:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017916.054553:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2178 00000100:00080000:3.0:1734017916.054555:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017916.054559:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017916.054563:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017916.054571:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332884352/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017916.054580:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017916.054583:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017916.054584:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017916.054587:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2178 00000100:00080000:3.0:1734017916.054590:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017916.054594:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017916.054599:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017916.054600:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332884096:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017916.054606:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332884480/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017916.054615:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017916.054618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017916.054620:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017916.054623:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2178 00000100:00080000:3.0:1734017916.054626:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017916.054629:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017916.054633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017916.054641:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332884608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017916.054649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017916.054652:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017916.054653:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017916.054657:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2178 00000100:00080000:3.0:1734017916.054660:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017916.054664:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:2.0:1734017916.054665:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017916.054668:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017916.054676:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332884736/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017916.054684:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734017916.054687:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017916/real 1734017916] req@ffff8800a73ce680 x1818247332884096/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017916.054688:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017916.054690:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017916.054693:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2178 00000100:00080000:3.0:1734017916.054696:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017916.054699:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017916.054703:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017916.054706:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332884096/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017916.054713:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332884864/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017916.054716:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017916.054720:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017916.054724:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332884096:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017916.054732:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332884224:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017916.054748:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017916.054755:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017916/real 1734017916] req@ffff8800a73cc380 x1818247332884224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017916.054766:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332884224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017916.054775:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017916.054777:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017916.054781:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332884224:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017916.054786:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332884352:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017916.054799:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017916.054805:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017916/real 1734017916] req@ffff8800a73cf100 x1818247332884352/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017916.054815:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332884352/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017916.054822:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017916.054825:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017916.054828:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332884352:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017916.054852:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332884480:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017916.054864:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017916.054870:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017916/real 1734017916] req@ffff8800a73cc700 x1818247332884480/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017916.054881:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332884480/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017916.054888:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017916.054890:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017916.054893:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332884480:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017916.054899:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332884608:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017916.054911:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017916.054917:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017916/real 1734017916] req@ffff8800a73cdf80 x1818247332884608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017916.054927:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332884608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017916.054935:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017916.054937:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017916.054940:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332884608:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017916.054946:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332884736:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017916.054957:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017916.054963:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017916/real 1734017916] req@ffff8800a73cf800 x1818247332884736/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017916.054972:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332884736/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017916.054979:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017916.054982:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017916.054985:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332884736:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017916.054990:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332884864:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017916.055002:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017916.055008:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017916/real 1734017916] req@ffff8800a73cfb80 x1818247332884864/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017916.055018:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332884864/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017916.055026:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017916.055028:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017916.055031:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332884864:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017916.536490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017917.538475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017918.540481:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017919.542532:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017920.544510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017921.062374:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017921.062386:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017921.062392:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017921.062406:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2183 00000100:00080000:3.0:1734017921.062419:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017921.062429:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017921.062477:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332884992/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017921.062516:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017921.062521:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017921.062523:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017921.062528:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2183 00000100:00080000:3.0:1734017921.062532:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017921.062536:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017921.062550:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017921.062563:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332885120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017921.062574:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017921.062578:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017921.062580:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017921.062584:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2183 00000100:00080000:3.0:1734017921.062586:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017921.062590:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017921.062596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017921.062605:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332885248/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017921.062613:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017921.062617:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017921.062618:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017921.062622:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2183 00000100:00080000:3.0:1734017921.062625:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017921.062628:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017921.062633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017921.062642:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332885376/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734017921.062643:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332884992:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017921.062651:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017921.062655:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017921.062658:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017921.062662:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2183 00000100:00080000:3.0:1734017921.062668:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017921.062672:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017921.062677:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017921.062686:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332885504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017921.062694:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017921.062697:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017921.062698:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017921.062703:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2183 00000100:00080000:3.0:1734017921.062708:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017921.062712:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017921.062717:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734017921.062718:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017921.062725:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332885632/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017921.062733:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017921.062737:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017921.062738:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734017921.062742:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017921/real 1734017921] req@ffff8800a73cdc00 x1818247332884992/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017921.062742:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2183 00000100:00080000:3.0:1734017921.062746:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017921.062749:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017921.062754:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017921.062762:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332884992/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017921.062763:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332885760/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734017921.062775:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017921.062779:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017921.062784:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332884992:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734017921.062793:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332885120:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017921.062810:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017921.062817:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017921/real 1734017921] req@ffff8800a73cfb80 x1818247332885120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017921.062828:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332885120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017921.062837:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017921.062840:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017921.062844:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332885120:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017921.062850:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332885248:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017921.062864:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017921.062870:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017921/real 1734017921] req@ffff8800a73cf800 x1818247332885248/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017921.062880:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332885248/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017921.062888:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017921.062890:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017921.062894:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332885248:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017921.062920:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332885376:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017921.062933:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017921.062939:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017921/real 1734017921] req@ffff8800a73cdf80 x1818247332885376/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017921.062950:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332885376/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017921.062957:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017921.062960:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017921.062962:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332885376:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017921.062969:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332885504:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734017921.062982:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017921.062988:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017921/real 1734017921] req@ffff8800a73cc700 x1818247332885504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017921.062998:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332885504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017921.063005:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017921.063008:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017921.063011:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332885504:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017921.063017:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332885632:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017921.063029:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017921.063035:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017921/real 1734017921] req@ffff8800a73cf100 x1818247332885632/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017921.063045:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332885632/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017921.063052:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017921.063054:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017921.063057:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332885632:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017921.063063:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332885760:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734017921.063074:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734017921.063080:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017921/real 1734017921] req@ffff8800a73cc380 x1818247332885760/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734017921.063091:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332885760/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734017921.063099:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734017921.063101:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734017921.063104:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332885760:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017921.546531:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017922.548497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017923.550505:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017924.552495:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017925.554482:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017926.070353:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017926.070362:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017926.070366:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017926.070377:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2188 00000100:00080000:3.0:1734017926.070386:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017926.070394:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017926.070430:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332885888/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017926.070462:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017926.070466:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017926.070468:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017926.070473:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2188 00000100:00080000:3.0:1734017926.070476:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017926.070480:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017926.070490:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017926.070500:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332886016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017926.070511:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017926.070514:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017926.070516:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017926.070519:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2188 00000100:00080000:3.0:1734017926.070524:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017926.070527:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017926.070532:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017926.070540:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332886144/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017926.070548:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017926.070551:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017926.070553:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017926.070556:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2188 00000100:00080000:3.0:1734017926.070558:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017926.070562:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017926.070566:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017926.070567:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332885888:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017926.070573:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332886272/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017926.070582:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017926.070585:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017926.070587:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017926.070590:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2188 00000100:00080000:3.0:1734017926.070593:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017926.070596:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017926.070600:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017926.070608:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332886400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017926.070623:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017926.070626:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017926.070628:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017926.070632:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2188 00000100:00080000:3.0:1734017926.070635:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017926.070638:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017926.070642:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017926.070649:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332886528/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017926.070657:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734017926.070658:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332886016:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734017926.070660:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017926.070662:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017926.070665:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2188 00000100:00080000:3.0:1734017926.070667:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017926.070671:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017926.070675:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734017926.070681:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332886144:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734017926.070683:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332886656/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734017926.070705:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332886272:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017926.070720:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332886400:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734017926.070735:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332886528:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734017926.070749:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332886656:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017926.556461:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017927.558510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017928.560486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734017929.076486:0:1956:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734017929.076534:0:1956:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734017929.076554:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017929.076594:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017929.076602:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017929.076609:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017929.076615:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017929.076623:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734017929.076630:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:3.0:1734017929.076709:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017926/real 1734017929] req@ffff8800a73cd880 x1818247332886272/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017929.076746:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332886272/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017929.076764:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017929.076769:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017929.076777:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332886272:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734017929.076787:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017926/real 1734017929] req@ffff8800a73cf480 x1818247332886400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017929.076809:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332886400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017929.076823:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017929.076828:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017929.076834:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332886400:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734017929.076845:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017926/real 1734017929] req@ffff8800a73cca80 x1818247332886528/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017929.076864:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332886528/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017929.076877:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017929.076881:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017929.076887:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332886528:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734017929.076897:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017926/real 1734017929] req@ffff8800a73cd500 x1818247332886656/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017929.076914:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332886656/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017929.076936:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017929.076940:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017929.076945:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332886656:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734017929.076955:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017926/real 1734017929] req@ffff8800a73ce680 x1818247332885888/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017929.076972:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332885888/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017929.076984:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017929.076988:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017929.076993:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332885888:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017929.077002:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017926/real 1734017929] req@ffff8800a73cce00 x1818247332886016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017929.077021:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332886016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017929.077032:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017929.077035:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017929.077040:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332886016:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734017929.077048:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017926/real 1734017929] req@ffff8800a73ced80 x1818247332886144/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017929.077066:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332886144/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017929.077088:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734017929.077092:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017929.077097:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332886144:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017929.562505:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017930.564529:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017931.078367:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017931.078376:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017931.078381:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017931.078391:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2193 00000100:00080000:3.0:1734017931.078401:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017931.078409:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017931.078444:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332886784/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017931.078478:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017931.078482:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017931.078484:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017931.078489:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2193 00000100:00080000:3.0:1734017931.078492:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017931.078496:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017931.078506:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017931.078517:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332886912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017931.078527:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017931.078530:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017931.078532:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017931.078536:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2193 00000100:00080000:3.0:1734017931.078539:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017931.078542:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017931.078547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017931.078555:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332887040/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017931.078564:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017931.078567:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017931.078568:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017931.078571:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2193 00000100:00080000:3.0:1734017931.078574:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017931.078577:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017931.078580:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332886784:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017931.078582:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017931.078589:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332887168/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017931.078598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017931.078601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017931.078603:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017931.078606:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2193 00000100:00080000:3.0:1734017931.078609:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017931.078612:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017931.078616:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017931.078624:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332887296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017931.078633:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017931.078636:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017931.078638:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017931.078642:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2193 00000100:00080000:3.0:1734017931.078644:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734017931.078646:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017931.078648:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017931.078652:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017931.078659:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332887424/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017931.078666:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017931.078667:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017931/real 1734017931] req@ffff8800a73ced80 x1818247332886784/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017931.078669:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017931.078671:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017931.078674:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2193 00000100:00080000:3.0:1734017931.078677:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017931.078680:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017931.078685:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017931.078686:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332886784/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017931.078692:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332887552/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017931.078697:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017931.078700:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017931.078705:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332886784:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017931.078712:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332886912:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017931.078730:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017931.078736:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017931/real 1734017931] req@ffff8800a73cce00 x1818247332886912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017931.078747:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332886912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017931.078755:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017931.078758:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017931.078761:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332886912:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017931.078767:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332887040:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017931.078782:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017931.078789:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017931/real 1734017931] req@ffff8800a73ce680 x1818247332887040/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017931.078800:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332887040/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017931.078807:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017931.078810:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017931.078813:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332887040:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017931.078837:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332887168:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017931.078850:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017931.078856:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017931/real 1734017931] req@ffff8800a73cd500 x1818247332887168/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017931.078866:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332887168/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017931.078873:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017931.078875:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017931.078878:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332887168:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017931.078884:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332887296:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017931.078896:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017931.078902:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017931/real 1734017931] req@ffff8800a73cca80 x1818247332887296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017931.078912:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332887296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017931.078927:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017931.078929:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017931.078933:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332887296:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017931.078938:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332887424:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017931.078949:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017931.078955:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017931/real 1734017931] req@ffff8800a73cf480 x1818247332887424/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017931.078965:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332887424/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017931.078972:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017931.078974:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017931.078977:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332887424:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017931.078983:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332887552:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017931.078995:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017931.079000:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017931/real 1734017931] req@ffff8800a73cd880 x1818247332887552/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017931.079010:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332887552/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017931.079017:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017931.079019:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017931.079022:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332887552:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017931.566457:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017932.568507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017933.570476:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017934.572465:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017935.574466:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017936.086343:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017936.086352:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017936.086357:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017936.086368:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2198 00000100:00080000:3.0:1734017936.086378:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017936.086385:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017936.086420:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332887680/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017936.086453:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017936.086457:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017936.086459:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017936.086463:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2198 00000100:00080000:3.0:1734017936.086466:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017936.086471:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017936.086481:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017936.086493:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332887808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017936.086503:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017936.086506:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017936.086508:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017936.086512:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2198 00000100:00080000:3.0:1734017936.086515:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017936.086518:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017936.086523:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017936.086531:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332887936/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017936.086540:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017936.086543:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017936.086545:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017936.086549:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2198 00000100:00080000:3.0:1734017936.086551:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017936.086555:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017936.086559:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017936.086562:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332887680:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017936.086567:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332888064/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017936.086576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017936.086579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017936.086581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017936.086584:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2198 00000100:00080000:3.0:1734017936.086587:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017936.086590:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017936.086594:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017936.086601:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332888192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017936.086610:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017936.086612:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017936.086614:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017936.086618:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2198 00000100:00080000:3.0:1734017936.086620:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017936.086624:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734017936.086627:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017936.086629:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017936.086637:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332888320/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017936.086645:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017936.086648:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734017936.086649:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017936/real 1734017936] req@ffff8800a73cea00 x1818247332887680/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017936.086650:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017936.086653:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2198 00000100:00080000:3.0:1734017936.086656:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017936.086659:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017936.086663:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017936.086668:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332887680/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017936.086671:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332888448/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017936.086680:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017936.086683:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017936.086688:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332887680:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017936.086696:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332887808:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017936.086713:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017936.086719:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017936/real 1734017936] req@ffff8800a73cd880 x1818247332887808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017936.086730:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332887808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017936.086738:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017936.086741:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017936.086744:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332887808:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017936.086750:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332887936:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017936.086763:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017936.086769:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017936/real 1734017936] req@ffff8800a73cf480 x1818247332887936/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017936.086787:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332887936/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017936.086795:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017936.086797:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017936.086800:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332887936:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017936.086824:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332888064:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017936.086836:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017936.086842:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017936/real 1734017936] req@ffff8800a73cca80 x1818247332888064/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017936.086852:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332888064/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017936.086860:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017936.086862:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017936.086865:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332888064:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017936.086871:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332888192:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017936.086882:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017936.086888:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017936/real 1734017936] req@ffff8800a73cd500 x1818247332888192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017936.086898:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332888192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017936.086906:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017936.086909:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017936.086912:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332888192:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017936.086918:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332888320:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017936.086930:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017936.086936:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017936/real 1734017936] req@ffff8800a73ce680 x1818247332888320/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017936.086946:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332888320/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017936.086953:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017936.086955:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017936.086959:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332888320:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017936.086964:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332888448:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017936.086976:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017936.086982:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017936/real 1734017936] req@ffff8800a73cce00 x1818247332888448/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017936.086992:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332888448/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017936.086999:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017936.087001:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017936.087004:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332888448:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017936.576506:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017937.578487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017938.580476:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017939.582455:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017940.584484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017941.094348:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017941.094356:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017941.094361:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017941.094371:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2203 00000100:00080000:3.0:1734017941.094381:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017941.094388:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017941.094424:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332888576/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017941.094457:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017941.094464:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017941.094467:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017941.094471:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2203 00000100:00080000:3.0:1734017941.094474:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017941.094478:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017941.094488:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017941.094498:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332888704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017941.094508:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017941.094512:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017941.094513:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017941.094517:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2203 00000100:00080000:3.0:1734017941.094520:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017941.094523:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017941.094528:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017941.094536:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332888832/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017941.094544:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017941.094547:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017941.094549:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017941.094552:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2203 00000100:00080000:3.0:1734017941.094554:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017941.094557:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017941.094562:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017941.094565:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332888576:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017941.094570:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332888960/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017941.094580:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017941.094583:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017941.094584:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017941.094588:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2203 00000100:00080000:3.0:1734017941.094590:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017941.094594:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017941.094598:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017941.094605:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332889088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017941.094614:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017941.094617:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017941.094618:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017941.094622:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2203 00000100:00080000:3.0:1734017941.094625:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017941.094628:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017941.094632:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017941.094634:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017941.094640:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332889216/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017941.094649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017941.094652:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017941.094654:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017941.094655:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017941/real 1734017941] req@ffff8800a73ced80 x1818247332888576/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017941.094657:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2203 00000100:00080000:3.0:1734017941.094660:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017941.094663:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017941.094668:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017941.094674:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332888576/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017941.094677:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332889344/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017941.094685:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017941.094688:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017941.094693:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332888576:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017941.094700:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332888704:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017941.094718:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017941.094724:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017941/real 1734017941] req@ffff8800a73cc380 x1818247332888704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017941.094735:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332888704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017941.094743:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017941.094746:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017941.094749:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332888704:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017941.094754:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332888832:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017941.094768:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017941.094773:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017941/real 1734017941] req@ffff8800a73cf100 x1818247332888832/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017941.094783:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332888832/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017941.094792:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017941.094794:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017941.094797:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332888832:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017941.094822:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332888960:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017941.094834:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017941.094840:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017941/real 1734017941] req@ffff8800a73cc700 x1818247332888960/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017941.094850:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332888960/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017941.094858:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017941.094860:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017941.094863:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332888960:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017941.094869:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332889088:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017941.094881:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017941.094886:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017941/real 1734017941] req@ffff8800a73cdf80 x1818247332889088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017941.094896:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332889088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017941.094904:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017941.094906:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017941.094909:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332889088:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017941.094916:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332889216:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017941.094928:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017941.094934:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017941/real 1734017941] req@ffff8800a73cf800 x1818247332889216/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017941.094944:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332889216/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017941.094952:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017941.094954:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017941.094957:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332889216:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017941.094962:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332889344:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017941.094974:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017941.094980:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017941/real 1734017941] req@ffff8800a73cfb80 x1818247332889344/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017941.094989:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332889344/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017941.095047:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017941.095049:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017941.095052:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332889344:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017941.586479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017942.588487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017943.590466:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017944.592491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017945.594470:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017946.102338:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017946.102347:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017946.102352:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017946.102361:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2208 00000100:00080000:3.0:1734017946.102371:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017946.102379:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017946.102416:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332889472/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017946.102448:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017946.102453:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017946.102455:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017946.102459:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2208 00000100:00080000:3.0:1734017946.102462:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017946.102466:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017946.102476:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017946.102489:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332889600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017946.102500:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017946.102503:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017946.102505:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017946.102508:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2208 00000100:00080000:3.0:1734017946.102511:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017946.102515:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017946.102519:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017946.102528:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332889728/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017946.102536:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017946.102539:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017946.102541:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017946.102544:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2208 00000100:00080000:3.0:1734017946.102546:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017946.102549:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017946.102554:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017946.102557:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332889472:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017946.102562:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332889856/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017946.102571:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017946.102574:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017946.102575:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017946.102579:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2208 00000100:00080000:3.0:1734017946.102581:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017946.102585:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017946.102589:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017946.102597:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332889984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017946.102606:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017946.102608:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017946.102610:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017946.102614:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2208 00000100:00080000:3.0:1734017946.102616:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017946.102620:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734017946.102621:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017946.102624:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017946.102631:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332890112/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017946.102639:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017946.102642:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734017946.102643:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017946/real 1734017946] req@ffff8800a73cdc00 x1818247332889472/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017946.102644:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017946.102647:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2208 00000100:00080000:3.0:1734017946.102650:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017946.102653:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017946.102657:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017946.102663:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332889472/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017946.102665:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332890240/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017946.102674:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017946.102677:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017946.102683:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332889472:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017946.102690:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332889600:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017946.102707:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017946.102714:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017946/real 1734017946] req@ffff8800a73cfb80 x1818247332889600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017946.102725:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332889600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017946.102733:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017946.102735:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017946.102739:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332889600:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017946.102744:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332889728:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017946.102758:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017946.102763:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017946/real 1734017946] req@ffff8800a73cf800 x1818247332889728/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017946.102773:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332889728/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017946.102781:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017946.102784:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017946.102787:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332889728:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017946.102811:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332889856:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017946.102823:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017946.102829:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017946/real 1734017946] req@ffff8800a73cdf80 x1818247332889856/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017946.102840:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332889856/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017946.102847:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017946.102849:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017946.102852:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332889856:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017946.102858:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332889984:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017946.102869:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017946.102875:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017946/real 1734017946] req@ffff8800a73cc700 x1818247332889984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017946.102885:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332889984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017946.102892:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017946.102894:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017946.102897:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332889984:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017946.102903:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332890112:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017946.102971:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017946.102979:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017946/real 1734017946] req@ffff8800a73cf100 x1818247332890112/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017946.102990:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332890112/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017946.102998:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017946.103000:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017946.103003:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332890112:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017946.103008:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332890240:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017946.103021:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017946.103026:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017946/real 1734017946] req@ffff8800a73cc380 x1818247332890240/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017946.103036:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332890240/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017946.103044:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017946.103046:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017946.103049:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332890240:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017946.596445:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017947.598448:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017948.600513:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017949.602481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017950.604454:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017951.110361:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017951.110370:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017951.110374:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017951.110384:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2213 00000100:00080000:3.0:1734017951.110395:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017951.110402:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017951.110437:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332890368/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017951.110471:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017951.110475:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017951.110477:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017951.110482:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2213 00000100:00080000:3.0:1734017951.110485:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017951.110489:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017951.110499:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017951.110510:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332890496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017951.110520:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017951.110523:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017951.110525:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017951.110528:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2213 00000100:00080000:3.0:1734017951.110532:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017951.110535:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017951.110540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017951.110548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332890624/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017951.110557:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017951.110559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017951.110561:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017951.110564:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2213 00000100:00080000:3.0:1734017951.110567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017951.110570:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017951.110574:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017951.110579:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332890368:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017951.110582:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332890752/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017951.110590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017951.110594:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017951.110595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017951.110598:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2213 00000100:00080000:3.0:1734017951.110601:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017951.110604:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017951.110608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017951.110616:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332890880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017951.110631:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017951.110634:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017951.110635:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017951.110639:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2213 00000100:00080000:3.0:1734017951.110642:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017951.110645:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734017951.110647:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017951.110649:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017951.110657:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332891008/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017951.110664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734017951.110668:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017951/real 1734017951] req@ffff8800a73ced80 x1818247332890368/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017951.110668:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017951.110669:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017951.110672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2213 00000100:00080000:3.0:1734017951.110675:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017951.110678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017951.110682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017951.110691:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332891136/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017951.110695:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332890368/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017951.110707:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017951.110710:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017951.110715:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332890368:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017951.110721:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332890496:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017951.110738:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017951.110745:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017951/real 1734017951] req@ffff8800a73cce00 x1818247332890496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017951.110756:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332890496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017951.110764:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017951.110766:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017951.110770:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332890496:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017951.110775:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332890624:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017951.110789:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017951.110795:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017951/real 1734017951] req@ffff8800a73ce680 x1818247332890624/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017951.110804:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332890624/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017951.110812:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017951.110816:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017951.110819:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332890624:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017951.110889:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332890752:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017951.110907:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017951.110914:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017951/real 1734017951] req@ffff8800a73cd500 x1818247332890752/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017951.110925:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332890752/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017951.110933:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017951.110935:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017951.110938:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332890752:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017951.110943:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332890880:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017951.110955:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017951.110961:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017951/real 1734017951] req@ffff8800a73cca80 x1818247332890880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017951.110970:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332890880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017951.110979:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017951.110981:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017951.110984:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332890880:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017951.110989:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332891008:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017951.111010:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017951.111016:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017951/real 1734017951] req@ffff8800a73cf480 x1818247332891008/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017951.111026:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332891008/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017951.111033:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017951.111035:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017951.111038:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332891008:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017951.111043:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332891136:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017951.111055:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017951.111060:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017951/real 1734017951] req@ffff8800a73cd880 x1818247332891136/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017951.111070:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332891136/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017951.111077:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017951.111080:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017951.111083:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332891136:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017951.606494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017952.608471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017953.610497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017954.612507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017955.614475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017956.118332:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017956.118341:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017956.118346:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017956.118357:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2218 00000100:00080000:3.0:1734017956.118367:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017956.118375:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017956.118410:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332891264/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017956.118443:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017956.118448:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017956.118450:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017956.118454:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2218 00000100:00080000:3.0:1734017956.118458:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017956.118462:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017956.118471:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017956.118483:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332891392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017956.118493:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017956.118496:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017956.118498:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017956.118502:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2218 00000100:00080000:3.0:1734017956.118505:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017956.118508:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017956.118512:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017956.118521:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332891520/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017956.118530:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017956.118534:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017956.118536:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017956.118539:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2218 00000100:00080000:3.0:1734017956.118542:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017956.118544:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332891264:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017956.118545:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017956.118549:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017956.118557:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332891648/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017956.118565:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017956.118568:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017956.118569:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017956.118573:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2218 00000100:00080000:3.0:1734017956.118575:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017956.118578:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017956.118583:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017956.118590:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332891776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017956.118598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017956.118601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017956.118603:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017956.118606:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2218 00000100:00080000:3.0:1734017956.118609:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734017956.118610:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017956.118613:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017956.118617:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017956.118625:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332891904/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017956.118631:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017956/real 1734017956] req@ffff8800a73cea00 x1818247332891264/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017956.118633:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017956.118636:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017956.118638:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017956.118641:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2218 00000100:00080000:3.0:1734017956.118644:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017956.118647:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017956.118651:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017956.118658:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332891264/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017956.118659:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332892032/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017956.118669:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017956.118672:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017956.118677:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332891264:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017956.118684:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332891392:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017956.118701:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017956.118708:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017956/real 1734017956] req@ffff8800a73cd880 x1818247332891392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017956.118718:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332891392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017956.118726:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017956.118729:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017956.118732:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332891392:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017956.118798:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332891520:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017956.118816:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017956.118823:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017956/real 1734017956] req@ffff8800a73cf480 x1818247332891520/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017956.118834:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332891520/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017956.118842:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017956.118845:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017956.118849:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332891520:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017956.118855:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332891648:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017956.118867:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017956.118873:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017956/real 1734017956] req@ffff8800a73cca80 x1818247332891648/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017956.118883:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332891648/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017956.118891:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017956.118893:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017956.118896:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332891648:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017956.118902:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332891776:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017956.118914:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017956.118919:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017956/real 1734017956] req@ffff8800a73cd500 x1818247332891776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017956.118929:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332891776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017956.118936:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017956.118938:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017956.118942:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332891776:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017956.118947:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332891904:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017956.118963:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017956.118969:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017956/real 1734017956] req@ffff8800a73ce680 x1818247332891904/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017956.118979:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332891904/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017956.118986:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017956.118988:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017956.118991:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332891904:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017956.118996:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332892032:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017956.119008:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017956.119014:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017956/real 1734017956] req@ffff8800a73cce00 x1818247332892032/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017956.119023:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332892032/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017956.119030:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017956.119032:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017956.119035:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332892032:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017956.616487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017957.618505:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017958.620513:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017959.622523:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017960.624489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017961.126360:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017961.126371:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017961.126377:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017961.126391:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2223 00000100:00080000:3.0:1734017961.126402:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017961.126411:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017961.126460:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332892160/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017961.126497:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017961.126505:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017961.126507:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017961.126512:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2223 00000100:00080000:3.0:1734017961.126516:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017961.126520:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017961.126532:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017961.126543:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332892288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017961.126554:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017961.126557:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017961.126559:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017961.126563:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2223 00000100:00080000:3.0:1734017961.126566:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017961.126570:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017961.126575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017961.126583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332892416/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017961.126591:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017961.126595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017961.126596:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017961.126600:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2223 00000100:00080000:3.0:1734017961.126602:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017961.126606:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017961.126611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017961.126612:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332892160:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017961.126620:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332892544/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017961.126629:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017961.126633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017961.126634:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017961.126638:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2223 00000100:00080000:3.0:1734017961.126641:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017961.126644:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017961.126648:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017961.126690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332892672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017961.126699:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017961.126702:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017961.126704:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017961.126708:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2223 00000100:00080000:3.0:1734017961.126711:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017961.126715:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017961.126720:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017961.126728:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332892800/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017961.126736:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1734017961.126737:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017961.126740:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017961.126742:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017961.126746:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2223 00000100:00080000:3.0:1734017961.126749:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017961.126752:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017961.126757:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017961.126762:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017961/real 1734017961] req@ffff8800a73ced80 x1818247332892160/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017961.126767:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332892928/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017961.126792:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332892160/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017961.126803:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017961.126806:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017961.126812:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332892160:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017961.126820:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332892288:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017961.126840:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017961.126847:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017961/real 1734017961] req@ffff8800a73cc380 x1818247332892288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017961.126858:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332892288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017961.126866:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017961.126868:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017961.126872:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332892288:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017961.126877:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332892416:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017961.126892:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017961.126898:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017961/real 1734017961] req@ffff8800a73cf100 x1818247332892416/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017961.126908:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332892416/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017961.126915:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017961.126919:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017961.126922:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332892416:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017961.126951:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332892544:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017961.126965:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017961.126971:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017961/real 1734017961] req@ffff8800a73cc700 x1818247332892544/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017961.126982:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332892544/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017961.126989:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017961.126991:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017961.126994:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332892544:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017961.127000:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332892672:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017961.127012:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017961.127017:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017961/real 1734017961] req@ffff8800a73cdf80 x1818247332892672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017961.127027:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332892672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017961.127034:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017961.127037:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017961.127040:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332892672:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017961.127046:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332892800:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017961.127062:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017961.127068:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017961/real 1734017961] req@ffff8800a73cf800 x1818247332892800/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017961.127079:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332892800/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017961.127087:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017961.127089:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017961.127092:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332892800:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017961.127097:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332892928:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017961.127109:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017961.127115:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017961/real 1734017961] req@ffff8800a73cfb80 x1818247332892928/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017961.127125:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332892928/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017961.127132:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017961.127134:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017961.127137:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332892928:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017961.626505:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017962.628495:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017963.630519:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017964.632472:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017965.634402:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017966.134369:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017966.134378:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017966.134382:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017966.134392:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2228 00000100:00080000:3.0:1734017966.134402:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017966.134410:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017966.134447:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332893056/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017966.134481:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017966.134485:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017966.134487:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017966.134492:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2228 00000100:00080000:3.0:1734017966.134495:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017966.134499:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017966.134509:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017966.134522:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332893184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017966.134533:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017966.134536:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017966.134538:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017966.134541:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2228 00000100:00080000:3.0:1734017966.134544:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017966.134547:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017966.134552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017966.134560:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332893312/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017966.134609:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017966.134613:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017966.134614:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017966.134616:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332893056:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017966.134618:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2228 00000100:00080000:3.0:1734017966.134621:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017966.134624:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017966.134629:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017966.134638:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332893440/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017966.134646:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017966.134650:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017966.134651:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017966.134655:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2228 00000100:00080000:3.0:1734017966.134658:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017966.134662:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017966.134666:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017966.134674:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332893568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017966.134682:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1734017966.134684:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017966.134685:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017966.134686:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017966.134691:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2228 00000100:00080000:3.0:1734017966.134694:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017966.134697:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017966.134701:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017966.134705:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017966/real 1734017966] req@ffff8800a73cdc00 x1818247332893056/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017966.134709:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332893696/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017966.134717:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017966.134720:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017966.134721:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017966.134725:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2228 00000100:00080000:3.0:1734017966.134728:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017966.134731:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017966.134734:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332893056/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734017966.134735:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017966.134743:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332893824/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017966.134744:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017966.134747:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017966.134752:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332893056:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017966.134760:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332893184:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017966.134777:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017966.134784:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017966/real 1734017966] req@ffff8800a73cfb80 x1818247332893184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017966.134795:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332893184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017966.134803:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017966.134806:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017966.134809:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332893184:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017966.134833:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332893312:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017966.134846:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017966.134852:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017966/real 1734017966] req@ffff8800a73cf800 x1818247332893312/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017966.134862:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332893312/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017966.134870:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017966.134873:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017966.134876:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332893312:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017966.134882:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332893440:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017966.134894:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017966.134900:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017966/real 1734017966] req@ffff8800a73cdf80 x1818247332893440/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017966.134910:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332893440/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017966.134917:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017966.134920:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017966.134923:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332893440:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017966.134928:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332893568:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017966.134940:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017966.134946:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017966/real 1734017966] req@ffff8800a73cc700 x1818247332893568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017966.134956:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332893568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017966.134963:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017966.134966:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017966.134970:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332893568:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017966.134976:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332893696:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017966.134991:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017966.134997:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017966/real 1734017966] req@ffff8800a73cf100 x1818247332893696/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017966.135007:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332893696/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017966.135015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017966.135017:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017966.135020:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332893696:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017966.135026:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332893824:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017966.135038:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017966.135043:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017966/real 1734017966] req@ffff8800a73cc380 x1818247332893824/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017966.135053:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332893824/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017966.135060:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017966.135063:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017966.135066:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332893824:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017966.636480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017967.638477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017968.640506:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017969.642497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017970.644534:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017971.142352:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017971.142363:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017971.142368:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017971.142381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2233 00000100:00080000:3.0:1734017971.142392:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017971.142402:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017971.142449:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332893952/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017971.142517:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017971.142522:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017971.142524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017971.142530:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2233 00000100:00080000:3.0:1734017971.142534:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017971.142539:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017971.142551:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017971.142564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332894080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017971.142575:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017971.142579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017971.142580:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017971.142584:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2233 00000100:00080000:3.0:1734017971.142589:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017971.142593:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017971.142598:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332893952:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734017971.142598:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017971.142607:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332894208/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017971.142617:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017971.142620:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017971.142622:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017971.142625:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2233 00000100:00080000:3.0:1734017971.142628:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017971.142632:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017971.142636:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017971.142645:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332894336/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017971.142653:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017971.142657:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017971.142659:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017971.142662:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2233 00000100:00080000:3.0:1734017971.142665:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017971.142669:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017971.142673:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017971.142681:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332894464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734017971.142686:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734017971.142696:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017971.142700:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017971.142701:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017971.142706:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2233 00000100:00080000:3.0:1734017971.142709:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017971.142712:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017971/real 1734017971] req@ffff8800a73ced80 x1818247332893952/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017971.142713:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017971.142717:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017971.142725:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332894592/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017971.142733:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017971.142737:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017971.142738:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017971.142742:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332893952/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734017996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017971.142742:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2233 00000100:00080000:3.0:1734017971.142744:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017971.142748:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017971.142752:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1734017971.142754:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017971.142758:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734017971.142762:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332894720/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017971.142763:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332893952:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017971.142770:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332894080:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017971.142790:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017971.142797:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017971/real 1734017971] req@ffff8800a73cce00 x1818247332894080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017971.142808:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332894080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017971.142816:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017971.142818:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017971.142822:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332894080:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017971.142850:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332894208:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017971.142865:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017971.142871:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017971/real 1734017971] req@ffff8800a73ce680 x1818247332894208/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017971.142881:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332894208/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017971.142890:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017971.142893:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017971.142896:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332894208:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017971.142902:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332894336:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017971.142915:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017971.142920:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017971/real 1734017971] req@ffff8800a73cd500 x1818247332894336/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017971.142930:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332894336/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017971.142938:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017971.142940:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017971.142944:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332894336:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017971.142950:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332894464:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017971.142963:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017971.142969:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017971/real 1734017971] req@ffff8800a73cca80 x1818247332894464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017971.142979:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332894464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734017996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017971.142988:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017971.142990:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017971.142993:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332894464:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017971.142998:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332894592:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017971.143014:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017971.143020:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017971/real 1734017971] req@ffff8800a73cf480 x1818247332894592/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017971.143031:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332894592/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017971.143038:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017971.143040:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017971.143043:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332894592:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017971.143048:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332894720:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017971.143061:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017971.143067:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017971/real 1734017971] req@ffff8800a73cd880 x1818247332894720/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017971.143077:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332894720/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734017996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017971.143085:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017971.143087:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017971.143090:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332894720:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017971.646550:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017972.648547:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017973.650394:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017974.652516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017975.654508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017976.150355:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017976.150367:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017976.150374:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017976.150388:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2238 00000100:00080000:3.0:1734017976.150445:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017976.150456:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017976.150503:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332894848/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017976.150541:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017976.150545:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017976.150547:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017976.150552:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2238 00000100:00080000:3.0:1734017976.150555:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017976.150560:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017976.150572:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017976.150586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332894976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017976.150596:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017976.150600:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017976.150602:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017976.150606:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2238 00000100:00080000:3.0:1734017976.150609:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017976.150613:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017976.150618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017976.150627:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332895104/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017976.150636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017976.150640:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017976.150642:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017976.150646:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2238 00000100:00080000:3.0:1734017976.150649:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734017976.150653:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332894848:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734017976.150653:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017976.150657:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017976.150665:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332895232/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017976.150673:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017976.150677:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017976.150678:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017976.150682:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2238 00000100:00080000:3.0:1734017976.150684:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017976.150688:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017976.150692:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017976.150701:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332895360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017976.150709:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017976.150712:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017976.150714:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017976.150718:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2238 00000100:00080000:3.0:1734017976.150730:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017976.150734:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734017976.150738:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017976.150739:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017976.150748:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332895488/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017976.150757:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017976.150760:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017976.150762:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017976.150764:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017976/real 1734017976] req@ffff8800a73cea00 x1818247332894848/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017976.150765:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2238 00000100:00080000:3.0:1734017976.150768:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017976.150772:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017976.150776:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017976.150785:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332895616/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017976.150795:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332894848/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017976.150806:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017976.150810:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017976.150815:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332894848:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017976.150823:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332894976:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017976.150841:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017976.150848:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017976/real 1734017976] req@ffff8800a73cd880 x1818247332894976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017976.150859:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332894976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017976.150867:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017976.150870:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017976.150873:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332894976:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017976.150899:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332895104:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017976.150913:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017976.150919:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017976/real 1734017976] req@ffff8800a73cf480 x1818247332895104/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017976.150929:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332895104/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017976.150937:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017976.150940:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017976.150943:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332895104:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017976.150950:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332895232:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017976.150962:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017976.150968:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017976/real 1734017976] req@ffff8800a73cca80 x1818247332895232/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017976.150978:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332895232/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017976.150986:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017976.150989:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017976.150992:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332895232:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017976.150997:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332895360:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017976.151010:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017976.151015:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017976/real 1734017976] req@ffff8800a73cd500 x1818247332895360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017976.151025:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332895360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017976.151033:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017976.151035:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017976.151038:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332895360:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017976.151043:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332895488:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017976.151060:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017976.151066:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017976/real 1734017976] req@ffff8800a73ce680 x1818247332895488/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017976.151076:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332895488/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017976.151083:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017976.151085:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017976.151088:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332895488:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017976.151094:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332895616:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017976.151106:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017976.151112:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017976/real 1734017976] req@ffff8800a73cce00 x1818247332895616/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017976.151122:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332895616/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017976.151129:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017976.151131:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017976.151134:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332895616:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017976.656483:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017977.658530:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017978.660531:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017979.662503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017980.664498:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017981.158397:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017981.158405:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017981.158409:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017981.158420:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2243 00000100:00080000:3.0:1734017981.158429:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017981.158437:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017981.158471:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332895744/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017981.158503:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017981.158510:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017981.158512:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017981.158516:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2243 00000100:00080000:3.0:1734017981.158519:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017981.158523:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017981.158533:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017981.158543:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332895872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017981.158553:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017981.158556:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017981.158558:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017981.158561:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2243 00000100:00080000:3.0:1734017981.158564:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017981.158567:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017981.158572:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017981.158580:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332896000/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017981.158589:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017981.158592:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017981.158593:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017981.158596:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2243 00000100:00080000:3.0:1734017981.158599:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017981.158602:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017981.158607:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017981.158611:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332895744:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017981.158615:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332896128/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017981.158624:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017981.158628:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017981.158629:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017981.158632:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2243 00000100:00080000:3.0:1734017981.158635:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017981.158638:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017981.158642:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017981.158650:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332896256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017981.158658:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017981.158661:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017981.158662:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017981.158666:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2243 00000100:00080000:3.0:1734017981.158669:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017981.158672:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734017981.158676:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017981.158676:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017981.158684:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332896384/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017981.158692:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017981.158695:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734017981.158697:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017981/real 1734017981] req@ffff8800a73ced80 x1818247332895744/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017981.158697:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017981.158701:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2243 00000100:00080000:3.0:1734017981.158704:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017981.158707:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017981.158711:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017981.158720:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332896512/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017981.158723:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332895744/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017981.158734:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017981.158737:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017981.158742:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332895744:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017981.158749:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332895872:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017981.158766:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017981.158773:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017981/real 1734017981] req@ffff8800a73cc380 x1818247332895872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017981.158784:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332895872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017981.158792:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017981.158794:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017981.158798:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332895872:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017981.158803:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332896000:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017981.158817:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017981.158823:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017981/real 1734017981] req@ffff8800a73cf100 x1818247332896000/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017981.158833:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332896000/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017981.158840:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017981.158843:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017981.158846:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332896000:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017981.158871:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332896128:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017981.158884:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017981.158890:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017981/real 1734017981] req@ffff8800a73cc700 x1818247332896128/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017981.158900:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332896128/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017981.158908:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017981.158910:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017981.158913:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332896128:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017981.158919:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332896256:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017981.158930:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017981.158936:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017981/real 1734017981] req@ffff8800a73cdf80 x1818247332896256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017981.158946:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332896256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017981.158953:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017981.158955:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017981.158958:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332896256:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017981.158964:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332896384:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017981.158979:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017981.158985:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017981/real 1734017981] req@ffff8800a73cf800 x1818247332896384/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017981.158996:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332896384/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017981.159003:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017981.159005:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017981.159008:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332896384:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017981.159013:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332896512:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017981.159025:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017981.159031:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017981/real 1734017981] req@ffff8800a73cfb80 x1818247332896512/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017981.159040:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332896512/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017981.159048:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017981.159050:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017981.159053:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332896512:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017981.666558:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017982.668500:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017983.670495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017984.672488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017985.674489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017986.166357:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017986.166366:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017986.166370:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017986.166380:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2248 00000100:00080000:3.0:1734017986.166391:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017986.166398:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017986.166435:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332896640/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017986.166469:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017986.166473:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017986.166475:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017986.166480:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2248 00000100:00080000:3.0:1734017986.166483:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017986.166487:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017986.166496:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017986.166509:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332896768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017986.166520:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017986.166523:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017986.166525:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017986.166528:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2248 00000100:00080000:3.0:1734017986.166531:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017986.166534:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017986.166539:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017986.166547:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332896896/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017986.166556:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017986.166559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017986.166560:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017986.166563:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2248 00000100:00080000:3.0:1734017986.166566:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017986.166569:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017986.166574:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017986.166580:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332896640:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017986.166582:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332897024/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017986.166590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017986.166594:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017986.166595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017986.166599:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2248 00000100:00080000:3.0:1734017986.166602:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017986.166605:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017986.166609:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017986.166618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332897152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017986.166626:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017986.166629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017986.166630:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017986.166634:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2248 00000100:00080000:3.0:1734017986.166637:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017986.166640:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017986.166644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734017986.166646:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734017986.166651:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332897280/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017986.166659:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017986.166662:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017986.166664:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017986.166667:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2248 00000100:00100000:0.0:1734017986.166668:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017986/real 1734017986] req@ffff8800a73cdc00 x1818247332896640/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017986.166670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017986.166673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017986.166677:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017986.166685:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332897408/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017986.166695:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332896640/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017986.166704:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017986.166708:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017986.166713:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332896640:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017986.166720:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332896768:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017986.166737:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017986.166743:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017986/real 1734017986] req@ffff8800a73cfb80 x1818247332896768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017986.166754:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332896768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017986.166762:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017986.166765:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017986.166768:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332896768:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017986.166774:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332896896:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017986.166787:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017986.166793:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017986/real 1734017986] req@ffff8800a73cf800 x1818247332896896/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017986.166803:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332896896/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017986.166810:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017986.166814:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017986.166817:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332896896:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017986.166846:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332897024:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017986.166859:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017986.166865:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017986/real 1734017986] req@ffff8800a73cdf80 x1818247332897024/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017986.166875:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332897024/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017986.166883:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017986.166885:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017986.166888:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332897024:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017986.166893:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332897152:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017986.166905:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017986.166910:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017986/real 1734017986] req@ffff8800a73cc700 x1818247332897152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017986.166920:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332897152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017986.166927:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017986.166929:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017986.166933:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332897152:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017986.166939:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332897280:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017986.166954:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017986.166960:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017986/real 1734017986] req@ffff8800a73cf100 x1818247332897280/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017986.166970:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332897280/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017986.166978:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017986.166980:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017986.166983:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332897280:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017986.166989:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332897408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017986.167001:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017986.167007:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017986/real 1734017986] req@ffff8800a73cc380 x1818247332897408/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017986.167016:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332897408/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017986.167024:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017986.167026:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017986.167029:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332897408:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017986.676451:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017987.678468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017988.680452:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017989.682489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017990.684478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017991.174362:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017991.174375:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017991.174381:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017991.174395:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2253 00000100:00080000:3.0:1734017991.174408:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017991.174418:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017991.174468:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332897536/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017991.174505:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017991.174510:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017991.174512:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017991.174518:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2253 00000100:00080000:3.0:1734017991.174521:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017991.174525:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017991.174536:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017991.174548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332897664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017991.174559:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017991.174562:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017991.174564:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017991.174568:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2253 00000100:00080000:3.0:1734017991.174573:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017991.174577:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017991.174582:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017991.174591:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332897792/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017991.174599:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017991.174602:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017991.174604:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017991.174608:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2253 00000100:00080000:3.0:1734017991.174610:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017991.174614:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017991.174618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017991.174620:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332897536:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017991.174627:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332897920/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017991.174636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017991.174639:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017991.174641:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017991.174644:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2253 00000100:00080000:3.0:1734017991.174647:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017991.174650:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017991.174655:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017991.174663:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332898048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017991.174679:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017991.174682:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017991.174684:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017991.174688:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2253 00000100:00080000:3.0:1734017991.174692:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017991.174695:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017991.174700:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017991.174708:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332898176/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017991.174716:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017991.174720:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017991.174721:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017991.174724:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2253 00000100:00080000:3.0:1734017991.174727:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017991.174730:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734017991.174734:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332897664:192.168.201.152@tcp:38:kworker.0 00000008:00100000:3.0:1734017991.174735:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017991.174745:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332898304/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734017991.174757:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332897792:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017991.174786:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332897920:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017991.174802:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332898048:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017991.174817:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332898176:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017991.174832:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332898304:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734017991.686505:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017992.688499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017993.690516:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1734017994.180499:0:1955:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:1.0:1734017994.180545:0:1955:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1734017994.180564:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017994.180600:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017994.180609:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017994.180617:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017994.180623:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017994.180630:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734017994.180636:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:0.0:1734017994.180726:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017991/real 1734017994] req@ffff8800a73cd500 x1818247332897920/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017994.180763:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332897920/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017994.180779:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017994.180785:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017994.180792:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332897920:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017994.180804:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017991/real 1734017994] req@ffff8800a73cca80 x1818247332898048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017994.180823:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332898048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017994.180837:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017994.180841:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017994.180847:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332898048:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017994.180858:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017991/real 1734017994] req@ffff8800a73cf480 x1818247332898176/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017994.180877:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332898176/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017994.180890:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017994.180895:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017994.180900:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332898176:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017994.180910:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017991/real 1734017994] req@ffff8800a73cd880 x1818247332898304/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017994.180927:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332898304/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017994.180939:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017994.180943:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017994.180948:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332898304:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017994.180957:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017991/real 1734017994] req@ffff8800a73ced80 x1818247332897536/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017994.180977:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332897536/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017994.180989:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017994.180993:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017994.180998:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332897536:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017994.181008:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017991/real 1734017994] req@ffff8800a73cce00 x1818247332897664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017994.181026:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332897664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017994.181049:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017994.181053:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017994.181059:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332897664:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017994.181068:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017991/real 1734017994] req@ffff8800a73ce680 x1818247332897792/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017994.181087:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332897792/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017994.181169:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017994.181174:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017994.181180:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332897792:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017994.692478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017995.694460:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734017996.182310:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017996.182319:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734017996.182323:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017996.182333:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2258 00000100:00080000:3.0:1734017996.182343:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017996.182351:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734017996.182387:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332898432/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017996.182421:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017996.182425:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017996.182427:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017996.182431:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2258 00000100:00080000:3.0:1734017996.182435:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017996.182438:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017996.182448:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017996.182459:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332898560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017996.182470:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017996.182473:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017996.182475:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017996.182479:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2258 00000100:00080000:3.0:1734017996.182481:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017996.182485:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017996.182489:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734017996.182497:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332898688/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017996.182506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017996.182510:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017996.182512:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017996.182515:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2258 00000100:00080000:3.0:1734017996.182518:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017996.182521:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017996.182526:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017996.182528:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332898432:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734017996.182533:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332898816/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017996.182542:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017996.182545:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734017996.182546:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017996.182550:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2258 00000100:00080000:3.0:1734017996.182552:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017996.182555:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017996.182559:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734017996.182567:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332898944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017996.182575:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017996.182578:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734017996.182579:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734017996.182583:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2258 00000100:00080000:3.0:1734017996.182586:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017996.182589:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734017996.182593:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734017996.182594:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734017996.182602:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332899072/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734017996.182610:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734017996.182613:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734017996.182614:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734017996.182615:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017996/real 1734017996] req@ffff8800a73cea00 x1818247332898432/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734017996.182618:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2258 00000100:00080000:3.0:1734017996.182620:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734017996.182623:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734017996.182627:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734017996.182634:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332898432/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734017996.182635:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332899200/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734017996.182644:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017996.182648:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017996.182652:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332898432:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734017996.182659:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332898560:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017996.182677:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017996.182684:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017996/real 1734017996] req@ffff8800a73ce680 x1818247332898560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017996.182695:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332898560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017996.182703:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017996.182705:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017996.182708:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332898560:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017996.182714:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332898688:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017996.182728:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017996.182734:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017996/real 1734017996] req@ffff8800a73cce00 x1818247332898688/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017996.182744:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332898688/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017996.182751:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017996.182753:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017996.182756:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332898688:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017996.182782:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332898816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017996.182794:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017996.182800:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017996/real 1734017996] req@ffff8800a73ced80 x1818247332898816/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017996.182811:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332898816/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017996.182825:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017996.182828:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017996.182831:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332898816:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017996.182836:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332898944:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734017996.182848:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017996.182854:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017996/real 1734017996] req@ffff8800a73cd880 x1818247332898944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017996.182863:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332898944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017996.182871:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017996.182873:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017996.182876:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332898944:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734017996.182882:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332899072:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017996.182894:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017996.182899:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017996/real 1734017996] req@ffff8800a73cf480 x1818247332899072/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017996.182909:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332899072/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017996.182916:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017996.182919:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017996.182922:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332899072:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734017996.182928:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332899200:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734017996.182941:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734017996.182946:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734017996/real 1734017996] req@ffff8800a73cca80 x1818247332899200/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734017996.182956:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332899200/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734017996.182963:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734017996.182966:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734017996.182969:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332899200:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734017996.696482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017997.698475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734017998.700467:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734017999.702486:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018000.704449:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018001.190355:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018001.190364:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018001.190369:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018001.190379:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2263 00000100:00080000:3.0:1734018001.190389:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018001.190397:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018001.190434:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332899328/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018001.190468:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018001.190475:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018001.190477:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018001.190481:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2263 00000100:00080000:3.0:1734018001.190485:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018001.190489:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018001.190500:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018001.190511:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332899456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018001.190521:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018001.190524:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018001.190526:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018001.190529:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2263 00000100:00080000:3.0:1734018001.190532:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018001.190535:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018001.190540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018001.190549:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332899584/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018001.190557:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018001.190560:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018001.190561:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018001.190565:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2263 00000100:00080000:3.0:1734018001.190567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018001.190571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734018001.190575:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332899328:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734018001.190576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018001.190583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332899712/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018001.190593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018001.190596:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018001.190597:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018001.190600:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2263 00000100:00080000:3.0:1734018001.190603:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018001.190606:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018001.190610:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018001.190618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332899840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018001.190626:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018001.190629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018001.190630:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018001.190634:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2263 00000100:00080000:3.0:1734018001.190637:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734018001.190640:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734018001.190640:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018001.190644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018001.190652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332899968/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018001.190660:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734018001.190661:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018001/real 1734018001] req@ffff8800a73cd500 x1818247332899328/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018001.190663:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018001.190665:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018001.190669:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2263 00000100:00080000:3.0:1734018001.190672:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018001.190675:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734018001.190680:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332899328/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734018001.190680:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018001.190689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332900096/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018001.190691:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018001.190694:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018001.190700:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332899328:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018001.190707:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332899456:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734018001.190723:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018001.190730:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018001/real 1734018001] req@ffff8800a73cc380 x1818247332899456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018001.190749:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332899456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018001.190756:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018001.190759:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018001.190762:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332899456:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018001.190768:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332899584:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018001.190781:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018001.190787:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018001/real 1734018001] req@ffff8800a73cf100 x1818247332899584/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018001.190797:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332899584/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018001.190804:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018001.190806:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018001.190810:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332899584:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018001.190838:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332899712:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018001.190850:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018001.190856:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018001/real 1734018001] req@ffff8800a73cc700 x1818247332899712/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018001.190866:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332899712/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018001.190873:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018001.190876:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018001.190879:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332899712:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018001.190885:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332899840:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734018001.190897:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018001.190903:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018001/real 1734018001] req@ffff8800a73cdf80 x1818247332899840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018001.190912:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332899840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018001.190920:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018001.190922:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018001.190925:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332899840:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018001.190931:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332899968:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018001.190942:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018001.190948:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018001/real 1734018001] req@ffff8800a73cf800 x1818247332899968/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018001.190958:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332899968/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018001.190965:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018001.190967:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018001.191024:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332899968:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018001.191030:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332900096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018001.191050:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018001.191057:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018001/real 1734018001] req@ffff8800a73cfb80 x1818247332900096/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018001.191068:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332900096/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018001.191076:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018001.191078:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018001.191081:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332900096:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018001.706485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018002.708474:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018003.710512:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018004.712468:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018005.714448:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018006.198356:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018006.198366:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018006.198370:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018006.198381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2268 00000100:00080000:3.0:1734018006.198391:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018006.198399:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018006.198437:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332900224/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018006.198471:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018006.198475:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018006.198477:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018006.198482:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2268 00000100:00080000:3.0:1734018006.198485:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018006.198489:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018006.198498:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018006.198511:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332900352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018006.198522:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018006.198526:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018006.198527:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018006.198531:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2268 00000100:00080000:3.0:1734018006.198533:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018006.198537:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018006.198541:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018006.198549:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332900480/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018006.198558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018006.198561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018006.198562:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018006.198565:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2268 00000100:00080000:3.0:1734018006.198568:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018006.198572:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734018006.198576:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332900224:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734018006.198576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018006.198583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332900608/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018006.198592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018006.198595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018006.198597:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018006.198601:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2268 00000100:00080000:3.0:1734018006.198604:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018006.198607:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018006.198611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018006.198619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332900736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018006.198627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018006.198630:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018006.198632:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018006.198635:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2268 00000100:00080000:3.0:1734018006.198638:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018006.198642:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734018006.198643:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734018006.198646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018006.198653:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332900864/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018006.198661:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018006.198664:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1734018006.198665:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018006/real 1734018006] req@ffff8800a73cdc00 x1818247332900224/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018006.198666:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018006.198669:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2268 00000100:00080000:3.0:1734018006.198672:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018006.198675:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018006.198679:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018006.198684:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332900224/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018006.198687:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332900992/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018006.198695:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018006.198699:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018006.198704:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332900224:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018006.198711:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332900352:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734018006.198728:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018006.198734:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018006/real 1734018006] req@ffff8800a73cfb80 x1818247332900352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018006.198745:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332900352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018006.198754:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018006.198757:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018006.198760:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332900352:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018006.198766:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332900480:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018006.198779:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018006.198785:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018006/real 1734018006] req@ffff8800a73cf800 x1818247332900480/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018006.198795:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332900480/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018006.198802:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018006.198805:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018006.198808:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332900480:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018006.198832:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332900608:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018006.198844:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018006.198850:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018006/real 1734018006] req@ffff8800a73cdf80 x1818247332900608/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018006.198861:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332900608/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018006.198868:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018006.198870:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018006.198873:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332900608:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018006.198880:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332900736:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734018006.198945:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018006.198954:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018006/real 1734018006] req@ffff8800a73cc700 x1818247332900736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018006.198965:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332900736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018006.198973:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018006.198975:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018006.198978:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332900736:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018006.198984:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332900864:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018006.198996:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018006.199002:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018006/real 1734018006] req@ffff8800a73cf100 x1818247332900864/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018006.199012:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332900864/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018006.199020:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018006.199022:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018006.199025:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332900864:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018006.199030:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332900992:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018006.199041:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018006.199047:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018006/real 1734018006] req@ffff8800a73cc380 x1818247332900992/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018006.199058:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332900992/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018006.199065:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018006.199068:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018006.199070:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332900992:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018006.716494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018007.718486:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018008.720454:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018009.722491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018010.724491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018011.206372:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018011.206385:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018011.206391:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018011.206407:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2273 00000100:00080000:3.0:1734018011.206420:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018011.206430:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018011.206479:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332901120/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018011.206516:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018011.206520:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018011.206522:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018011.206528:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2273 00000100:00080000:3.0:1734018011.206531:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018011.206535:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018011.206547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018011.206558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332901248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018011.206569:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018011.206573:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018011.206575:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018011.206579:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2273 00000100:00080000:3.0:1734018011.206583:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018011.206587:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018011.206592:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018011.206601:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332901376/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018011.206610:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018011.206613:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018011.206614:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018011.206618:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2273 00000100:00080000:3.0:1734018011.206621:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018011.206625:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734018011.206629:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332901120:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734018011.206629:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018011.206637:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332901504/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018011.206646:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018011.206650:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018011.206651:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018011.206655:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2273 00000100:00080000:3.0:1734018011.206657:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018011.206661:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018011.206665:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018011.206674:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332901632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018011.206689:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018011.206692:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018011.206694:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018011.206698:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2273 00000100:00080000:3.0:1734018011.206703:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018011.206706:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018011.206711:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734018011.206716:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734018011.206719:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332901760/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018011.206727:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018011.206730:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018011.206732:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018011.206735:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2273 00000100:00080000:3.0:1734018011.206738:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734018011.206741:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018011/real 1734018011] req@ffff8800a73cd500 x1818247332901120/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018011.206742:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018011.206746:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018011.206757:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332901888/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018011.206764:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332901120/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018011.206776:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018011.206779:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018011.206785:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332901120:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018011.206793:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332901248:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734018011.206861:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018011.206870:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018011/real 1734018011] req@ffff8800a73cca80 x1818247332901248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018011.206881:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332901248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018011.206890:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018011.206893:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018011.206897:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332901248:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018011.206904:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332901376:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018011.206919:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018011.206926:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018011/real 1734018011] req@ffff8800a73cf480 x1818247332901376/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018011.206936:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332901376/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018011.206944:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018011.206947:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018011.206950:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332901376:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018011.206979:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332901504:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018011.206992:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018011.206998:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018011/real 1734018011] req@ffff8800a73cd880 x1818247332901504/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018011.207009:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332901504/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018011.207017:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018011.207019:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018011.207022:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332901504:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018011.207027:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332901632:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734018011.207043:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018011.207049:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018011/real 1734018011] req@ffff8800a73ced80 x1818247332901632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018011.207059:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332901632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018011.207067:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018011.207070:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018011.207073:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332901632:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018011.207078:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332901760:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018011.207090:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018011.207096:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018011/real 1734018011] req@ffff8800a73cce00 x1818247332901760/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018011.207105:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332901760/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018011.207112:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018011.207114:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018011.207117:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332901760:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018011.207123:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332901888:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018011.207135:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018011.207141:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018011/real 1734018011] req@ffff8800a73ce680 x1818247332901888/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018011.207158:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332901888/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018011.207166:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018011.207168:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018011.207171:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332901888:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018011.726545:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018012.728486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018013.730466:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018014.732468:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018015.734457:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018016.214346:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018016.214355:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018016.214359:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018016.214370:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2278 00000100:00080000:3.0:1734018016.214380:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018016.214388:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018016.214422:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247332902016/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018016.214456:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018016.214461:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018016.214463:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018016.214467:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2278 00000100:00080000:3.0:1734018016.214471:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018016.214475:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018016.214484:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018016.214496:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332902144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018016.214506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018016.214509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018016.214511:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018016.214514:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2278 00000100:00080000:3.0:1734018016.214517:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018016.214520:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018016.214525:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018016.214533:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332902272/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018016.214541:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018016.214545:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018016.214547:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018016.214551:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2278 00000100:00080000:3.0:1734018016.214553:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018016.214557:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018016.214561:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018016.214567:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332902016:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734018016.214568:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332902400/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018016.214577:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018016.214581:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018016.214582:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018016.214585:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2278 00000100:00080000:3.0:1734018016.214588:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018016.214591:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018016.214595:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018016.214603:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332902528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018016.214611:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018016.214614:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018016.214615:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018016.214619:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2278 00000100:00080000:3.0:1734018016.214623:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018016.214626:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018016.214631:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734018016.214632:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734018016.214639:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332902656/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018016.214647:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018016.214650:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018016.214652:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734018016.214653:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018016/real 1734018016] req@ffff8800a73cea00 x1818247332902016/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018016.214655:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2278 00000100:00080000:3.0:1734018016.214658:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018016.214661:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018016.214665:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018016.214673:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247332902016/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018016.214673:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332902784/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018016.214683:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018016.214687:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018016.214692:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332902016:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018016.214699:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332902144:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734018016.214749:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018016.214757:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018016/real 1734018016] req@ffff8800a73ce680 x1818247332902144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018016.214769:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332902144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018016.214777:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018016.214781:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018016.214784:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332902144:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018016.214791:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332902272:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018016.214806:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018016.214812:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018016/real 1734018016] req@ffff8800a73cce00 x1818247332902272/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018016.214822:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332902272/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018016.214830:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018016.214832:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018016.214835:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332902272:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018016.214860:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332902400:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018016.214873:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018016.214879:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018016/real 1734018016] req@ffff8800a73ced80 x1818247332902400/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018016.214889:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332902400/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018016.214897:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018016.214899:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018016.214902:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332902400:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018016.214907:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332902528:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734018016.214922:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018016.214928:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018016/real 1734018016] req@ffff8800a73cd880 x1818247332902528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018016.214938:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332902528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018016.214946:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018016.214948:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018016.214951:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332902528:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018016.214957:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332902656:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018016.214968:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018016.214974:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018016/real 1734018016] req@ffff8800a73cf480 x1818247332902656/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018016.214984:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332902656/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018016.214991:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018016.214993:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018016.214995:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332902656:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018016.215001:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332902784:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018016.215013:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018016.215019:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018016/real 1734018016] req@ffff8800a73cca80 x1818247332902784/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018016.215035:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332902784/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018016.215043:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018016.215045:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018016.215048:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332902784:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018016.736474:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018017.738446:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018018.740478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018019.742447:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018020.744478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018021.222350:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018021.222358:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018021.222363:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018021.222373:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2283 00000100:00080000:3.0:1734018021.222383:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018021.222390:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018021.222423:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332902912/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018021.222457:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018021.222463:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018021.222466:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018021.222472:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2283 00000100:00080000:3.0:1734018021.222477:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018021.222482:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018021.222494:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018021.222511:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332903040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018021.222527:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018021.222532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018021.222534:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018021.222539:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2283 00000100:00080000:3.0:1734018021.222544:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018021.222551:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018021.222559:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018021.222566:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332902912:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734018021.222574:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332903168/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018021.222589:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018021.222594:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018021.222597:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018021.222602:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2283 00000100:00080000:3.0:1734018021.222605:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018021.222609:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018021.222616:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018021.222626:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332903296/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734018021.222663:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734018021.222673:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018021.222677:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018021.222679:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018021.222682:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2283 00000100:00100000:0.0:1734018021.222686:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018021/real 1734018021] req@ffff8800a73cd500 x1818247332902912/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018021.222686:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018021.222689:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018021.222695:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018021.222704:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332903424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018021.222705:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332902912/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018021.222713:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018021.222716:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734018021.222716:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018021.222718:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018021.222719:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734018021.222722:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2283 00000100:00100000:0.0:1734018021.222724:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332902912:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734018021.222724:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018021.222728:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734018021.222732:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332903040:192.168.201.152@tcp:38:kworker.0 00000008:00100000:3.0:1734018021.222732:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018021.222740:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332903552/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018021.222748:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018021.222751:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1734018021.222752:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734018021.222753:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018021.222757:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2283 00000100:00100000:0.0:1734018021.222759:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018021/real 1734018021] req@ffff8800a73cc380 x1818247332903040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018021.222760:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018021.222763:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018021.222768:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018021.222771:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332903040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018021.222777:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332903680/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018021.222779:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018021.222782:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018021.222786:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332903040:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018021.222808:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332903168:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018021.222822:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018021.222828:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018021/real 1734018021] req@ffff8800a73cf100 x1818247332903168/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018021.222839:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332903168/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018021.222847:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018021.222849:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018021.222852:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332903168:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018021.222857:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332903296:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018021.222868:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018021.222874:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018021/real 1734018021] req@ffff8800a73cc700 x1818247332903296/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018021.222884:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332903296/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018021.222891:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018021.222894:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018021.222897:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332903296:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018021.222903:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332903424:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734018021.222918:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018021.222924:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018021/real 1734018021] req@ffff8800a73cdf80 x1818247332903424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018021.222935:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332903424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018021.222942:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018021.222945:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018021.222948:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332903424:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018021.222953:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332903552:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018021.222965:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018021.222971:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018021/real 1734018021] req@ffff8800a73cf800 x1818247332903552/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018021.222981:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332903552/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018021.222988:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018021.222990:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018021.222993:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332903552:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018021.222998:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332903680:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018021.223010:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018021.223016:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018021/real 1734018021] req@ffff8800a73cfb80 x1818247332903680/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018021.223033:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332903680/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018021.223040:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018021.223042:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018021.223045:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332903680:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018021.746462:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018022.748360:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018023.750467:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018024.752358:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018025.758523:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018026.230325:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018026.230332:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018026.230335:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018026.230342:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2288 00000100:00080000:3.0:1734018026.230350:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018026.230355:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018026.230380:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247332903808/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018026.230402:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018026.230405:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018026.230406:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018026.230409:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2288 00000100:00080000:3.0:1734018026.230410:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018026.230413:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018026.230419:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018026.230427:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247332903936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018026.230433:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018026.230435:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018026.230436:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018026.230438:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2288 00000100:00080000:3.0:1734018026.230439:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018026.230441:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018026.230443:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018026.230448:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247332904064/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018026.230452:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018026.230454:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018026.230455:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018026.230456:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2288 00000100:00080000:3.0:1734018026.230458:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018026.230459:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018026.230462:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018026.230466:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247332904192/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018026.230470:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018026.230472:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018026.230473:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018026.230474:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2288 00000100:00080000:3.0:1734018026.230476:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018026.230478:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018026.230480:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018026.230484:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332903808:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734018026.230484:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247332904320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018026.230489:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018026.230491:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018026.230491:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018026.230494:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2288 00000100:00080000:3.0:1734018026.230495:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018026.230496:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018026.230499:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018026.230502:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247332904448/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018026.230506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018026.230508:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018026.230509:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018026.230511:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2288 00000100:00080000:3.0:1734018026.230513:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018026.230515:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018026.230517:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018026.230521:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247332904576/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734018026.230529:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018026.230555:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018026/real 1734018026] req@ffff8800a73cdc00 x1818247332903808/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018026.230567:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247332903808/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018026.230572:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018026.230574:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018026.230577:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332903808:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018026.230580:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332903936:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734018026.230591:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018026.230595:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018026/real 1734018026] req@ffff8800a73cfb80 x1818247332903936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018026.230600:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247332903936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018026.230604:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018026.230606:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018026.230608:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332903936:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018026.230611:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332904064:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018026.230618:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018026.230621:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018026/real 1734018026] req@ffff8800a73cf800 x1818247332904064/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018026.230626:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247332904064/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018026.230630:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018026.230632:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018026.230634:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332904064:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018026.230636:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332904192:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018026.230642:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018026.230645:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018026/real 1734018026] req@ffff8800a73cdf80 x1818247332904192/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018026.230650:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247332904192/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018026.230654:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018026.230655:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018026.230657:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332904192:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018026.230672:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332904320:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734018026.230681:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018026.230684:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018026/real 1734018026] req@ffff8800a73cc700 x1818247332904320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018026.230689:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247332904320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018026.230693:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018026.230694:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018026.230696:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332904320:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018026.230698:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332904448:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018026.230705:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018026.230707:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018026/real 1734018026] req@ffff8800a73cf100 x1818247332904448/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018026.230712:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247332904448/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018026.230716:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018026.230717:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018026.230719:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332904448:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018026.230721:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332904576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018026.230727:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018026.230730:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018026/real 1734018026] req@ffff8800a73cc380 x1818247332904576/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018026.230738:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247332904576/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018026.230743:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018026.230744:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018026.230745:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332904576:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018026.760409:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018027.762493:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018028.764451:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018029.766461:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018030.768462:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018031.238255:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018031.238262:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018031.238264:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018031.238271:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2293 00000100:00080000:3.0:1734018031.238278:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018031.238282:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018031.238306:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247332904704/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018031.238322:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018031.238324:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018031.238324:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018031.238327:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2293 00000100:00080000:3.0:1734018031.238329:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018031.238331:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018031.238336:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018031.238341:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247332904832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018031.238346:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018031.238347:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018031.238348:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018031.238350:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2293 00000100:00080000:3.0:1734018031.238352:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018031.238353:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734018031.238356:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332904704:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734018031.238356:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018031.238360:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247332904960/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018031.238364:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018031.238365:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018031.238366:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018031.238367:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2293 00000100:00080000:3.0:1734018031.238368:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018031.238370:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018031.238372:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018031.238375:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247332905088/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018031.238379:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018031.238381:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018031.238381:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018031.238383:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2293 00000100:00080000:3.0:1734018031.238384:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018031.238386:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018031.238388:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018031.238391:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247332905216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018031.238398:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018031.238399:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018031.238400:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1734018031.238401:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734018031.238402:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2293 00000100:00080000:3.0:1734018031.238403:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018031.238405:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018031.238406:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018031.238410:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247332905344/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018031.238413:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018031.238415:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018031.238415:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734018031.238416:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018031/real 1734018031] req@ffff8800a73cd500 x1818247332904704/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018031.238417:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2293 00000100:00080000:3.0:1734018031.238418:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018031.238419:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018031.238421:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018031.238426:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247332905472/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734018031.238429:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247332904704/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018031.238436:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018031.238438:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018031.238441:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332904704:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018031.238446:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332904832:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734018031.238471:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018031.238477:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018031/real 1734018031] req@ffff8800a73cca80 x1818247332904832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018031.238486:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247332904832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018031.238492:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018031.238494:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018031.238497:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332904832:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018031.238513:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332904960:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018031.238525:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018031.238529:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018031/real 1734018031] req@ffff8800a73cf480 x1818247332904960/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018031.238537:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247332904960/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018031.238542:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018031.238544:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018031.238546:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332904960:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018031.238549:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332905088:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018031.238558:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018031.238562:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018031/real 1734018031] req@ffff8800a73cd880 x1818247332905088/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018031.238569:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247332905088/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018031.238575:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018031.238576:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018031.238578:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332905088:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018031.238582:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332905216:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734018031.238593:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018031.238597:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018031/real 1734018031] req@ffff8800a73ced80 x1818247332905216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018031.238604:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247332905216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018031.238609:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018031.238611:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018031.238613:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332905216:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018031.238617:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332905344:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018031.238626:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018031.238630:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018031/real 1734018031] req@ffff8800a73cce00 x1818247332905344/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018031.238638:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247332905344/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018031.238643:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018031.238644:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018031.238646:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332905344:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018031.238650:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332905472:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734018031.238661:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018031.238665:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018031/real 1734018031] req@ffff8800a73ce680 x1818247332905472/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018031.238678:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247332905472/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018031.238684:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018031.238685:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018031.238687:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332905472:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018031.770365:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018032.772374:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018033.774392:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018034.776346:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018035.778427:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018036.246409:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018036.246418:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018036.246422:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018036.246431:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2298 00000100:00080000:0.0:1734018036.246440:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018036.246447:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018036.246473:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332905600/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018036.246495:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018036.246497:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018036.246498:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018036.246501:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2298 00000100:00080000:0.0:1734018036.246503:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018036.246505:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018036.246511:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018036.246517:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332905728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018036.246526:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018036.246528:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018036.246529:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018036.246531:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2298 00000100:00080000:0.0:1734018036.246532:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018036.246534:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018036.246537:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018036.246541:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332905856/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018036.246545:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018036.246547:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018036.246548:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018036.246550:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2298 00000100:00080000:0.0:1734018036.246551:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018036.246553:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:2.0:1734018036.246553:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332905600:192.168.201.152@tcp:250:kworker.0 00000008:00100000:0.0:1734018036.246555:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018036.246560:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332905984/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018036.246564:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018036.246566:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018036.246566:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018036.246568:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2298 00000100:00080000:0.0:1734018036.246570:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018036.246572:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018036.246574:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018036.246578:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332906112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018036.246582:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018036.246584:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018036.246584:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018036.246586:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2298 00000100:00080000:0.0:1734018036.246588:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018036.246589:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018036.246591:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018036.246595:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332906240/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018036.246599:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018036.246601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018036.246602:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018036.246603:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2298 00000800:00000100:2.0:1734018036.246603:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018036.246605:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018036.246606:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018036.246608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018036.246612:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332906368/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018036.246621:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018036/real 1734018036] req@ffff8800a705d880 x1818247332905600/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018036.246636:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332905600/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018036.246645:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018036.246648:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018036.246651:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332905600:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734018036.246658:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332905728:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734018036.246676:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018036.246683:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018036/real 1734018036] req@ffff8800a705d500 x1818247332905728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018036.246692:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332905728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018036.246699:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018036.246701:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018036.246704:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332905728:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734018036.246724:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332905856:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734018036.246739:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018036.246745:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018036/real 1734018036] req@ffff8800a705e300 x1818247332905856/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018036.246754:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332905856/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018036.246762:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018036.246764:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018036.246767:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332905856:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734018036.246772:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332905984:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734018036.246785:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018036.246791:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018036/real 1734018036] req@ffff8800a705c000 x1818247332905984/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018036.246805:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332905984/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018036.246812:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018036.246814:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018036.246817:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332905984:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734018036.246822:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332906112:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734018036.246836:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018036.246843:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018036/real 1734018036] req@ffff8800a705e680 x1818247332906112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018036.246852:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332906112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018036.246859:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018036.246861:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018036.246863:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332906112:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734018036.246868:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332906240:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734018036.246881:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018036.246887:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018036/real 1734018036] req@ffff8800a705ca80 x1818247332906240/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018036.246896:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332906240/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018036.246903:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018036.246905:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018036.246908:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332906240:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734018036.246913:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332906368:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734018036.246927:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018036.246933:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018036/real 1734018036] req@ffff8800a705f800 x1818247332906368/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018036.246942:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332906368/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018036.246948:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018036.246949:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018036.246952:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332906368:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018036.780412:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018037.782417:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018038.784428:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018039.786431:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018040.788352:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018041.254328:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018041.254336:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018041.254339:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018041.254346:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2303 00000100:00080000:0.0:1734018041.254353:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018041.254358:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018041.254387:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332906496/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018041.254409:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018041.254411:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018041.254412:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018041.254415:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2303 00000100:00080000:0.0:1734018041.254417:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018041.254419:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018041.254426:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018041.254432:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332906624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018041.254437:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018041.254439:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018041.254440:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018041.254442:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2303 00000100:00080000:0.0:1734018041.254444:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018041.254445:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018041.254449:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018041.254454:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332906752/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018041.254459:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018041.254461:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018041.254462:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734018041.254463:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332906496:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018041.254464:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2303 00000100:00080000:0.0:1734018041.254466:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018041.254468:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018041.254470:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018041.254474:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332906880/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018041.254478:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018041.254480:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018041.254481:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018041.254482:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2303 00000100:00080000:0.0:1734018041.254484:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018041.254485:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018041.254487:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018041.254491:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332907008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018041.254496:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018041.254497:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018041.254498:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018041.254500:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2303 00000100:00080000:0.0:1734018041.254502:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018041.254504:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018041.254506:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018041.254510:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332907136/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734018041.254513:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018041.254514:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018041.254515:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018041.254516:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018041.254518:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2303 00000100:00080000:0.0:1734018041.254520:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018041.254521:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018041.254524:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018041.254528:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332907264/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018041.254530:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018041/real 1734018041] req@ffff8800a705fb80 x1818247332906496/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018041.254544:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332906496/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018041.254552:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018041.254555:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018041.254559:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332906496:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734018041.254567:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332906624:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734018041.254583:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018041.254589:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018041/real 1734018041] req@ffff8800a705ea00 x1818247332906624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018041.254599:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332906624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018041.254606:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018041.254608:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018041.254611:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332906624:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734018041.254640:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332906752:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734018041.254655:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018041.254662:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018041/real 1734018041] req@ffff8800a705f800 x1818247332906752/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018041.254672:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332906752/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018041.254678:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018041.254680:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018041.254683:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332906752:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734018041.254687:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332906880:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734018041.254700:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018041.254705:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018041/real 1734018041] req@ffff8800a705ca80 x1818247332906880/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018041.254714:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332906880/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018041.254720:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018041.254722:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018041.254725:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332906880:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734018041.254729:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332907008:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734018041.254742:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018041.254748:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018041/real 1734018041] req@ffff8800a705e680 x1818247332907008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018041.254757:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332907008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018041.254763:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018041.254765:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018041.254768:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332907008:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734018041.254772:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332907136:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734018041.254785:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018041.254790:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018041/real 1734018041] req@ffff8800a705c000 x1818247332907136/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018041.254798:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332907136/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018041.254804:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018041.254806:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018041.254809:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332907136:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734018041.254814:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332907264:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734018041.254826:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018041.254831:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018041/real 1734018041] req@ffff8800a705e300 x1818247332907264/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018041.254840:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332907264/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018041.254846:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018041.254848:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018041.254850:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332907264:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018041.790434:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018042.792376:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018043.794499:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018044.796559:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018045.798545:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018046.262342:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018046.262351:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018046.262356:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018046.262365:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2308 00000100:00080000:0.0:1734018046.262373:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018046.262380:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018046.262408:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332907392/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018046.262445:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018046.262448:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018046.262450:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018046.262453:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2308 00000100:00080000:0.0:1734018046.262456:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018046.262459:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018046.262467:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018046.262478:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332907520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018046.262488:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018046.262491:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018046.262493:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018046.262497:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2308 00000100:00080000:0.0:1734018046.262499:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018046.262503:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018046.262507:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018046.262513:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332907392:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018046.262516:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332907648/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018046.262525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018046.262528:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018046.262529:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018046.262533:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2308 00000100:00080000:0.0:1734018046.262536:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018046.262539:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018046.262543:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018046.262551:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332907776/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018046.262559:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018046.262562:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018046.262563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1734018046.262565:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018046.262566:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2308 00000100:00080000:0.0:1734018046.262569:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018046.262571:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018046.262576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018046.262583:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018046/real 1734018046] req@ffff8800a705d500 x1818247332907392/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018046.262584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332907904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018046.262591:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018046.262595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018046.262596:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734018046.262599:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332907392/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018046.262600:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2308 00000100:00080000:0.0:1734018046.262603:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018046.262606:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:2.0:1734018046.262609:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1734018046.262610:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1734018046.262612:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018046.262617:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332907392:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018046.262618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332908032/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018046.262624:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332907520:192.168.201.152@tcp:38:kworker.0 00000100:00080000:0.0:1734018046.262627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018046.262631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018046.262632:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018046.262635:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2308 00000100:00080000:0.0:1734018046.262639:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734018046.262639:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018046.262642:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018046.262646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018046.262646:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018046/real 1734018046] req@ffff8800a705d880 x1818247332907520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018046.262657:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332908160/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018046.262657:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332907520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018046.262665:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018046.262667:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018046.262671:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332907520:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734018046.262690:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332907648:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734018046.262706:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018046.262712:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018046/real 1734018046] req@ffff8800a705ed80 x1818247332907648/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018046.262723:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332907648/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018046.262731:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018046.262733:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018046.262736:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332907648:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734018046.262742:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332907776:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734018046.262753:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018046.262759:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018046/real 1734018046] req@ffff8800a705c700 x1818247332907776/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018046.262769:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332907776/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018046.262777:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018046.262779:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018046.262782:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332907776:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734018046.262787:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332907904:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734018046.262798:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018046.262803:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018046/real 1734018046] req@ffff8800a705df80 x1818247332907904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018046.262813:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332907904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018046.262821:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018046.262823:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018046.262826:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332907904:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734018046.262831:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332908032:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734018046.262842:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018046.262847:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018046/real 1734018046] req@ffff8800a705dc00 x1818247332908032/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018046.262857:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332908032/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018046.262865:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018046.262867:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018046.262871:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332908032:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734018046.262876:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332908160:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734018046.262887:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018046.262892:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018046/real 1734018046] req@ffff8800a705f100 x1818247332908160/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018046.262902:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332908160/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018046.262910:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018046.262912:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018046.262915:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332908160:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018046.800494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018047.802606:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018048.804437:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018049.806585:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018050.808397:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018051.270305:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018051.270312:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018051.270315:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018051.270322:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2313 00000100:00080000:0.0:1734018051.270330:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018051.270335:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018051.270360:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332908288/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018051.270385:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018051.270387:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018051.270388:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018051.270391:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2313 00000100:00080000:0.0:1734018051.270392:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018051.270394:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018051.270400:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018051.270406:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332908416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018051.270412:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018051.270414:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018051.270415:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018051.270417:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2313 00000100:00080000:0.0:1734018051.270418:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018051.270420:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018051.270423:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018051.270429:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332908544/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018051.270433:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018051.270434:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018051.270435:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018051.270437:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2313 00000100:00080000:0.0:1734018051.270438:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018051.270440:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018051.270443:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018051.270447:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332908672/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018051.270447:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332908288:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018051.270452:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018051.270454:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018051.270454:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018051.270456:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2313 00000100:00080000:0.0:1734018051.270458:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018051.270460:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018051.270462:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018051.270466:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332908800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018051.270470:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018051.270472:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018051.270472:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018051.270474:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2313 00000100:00080000:0.0:1734018051.270476:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018051.270477:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018051.270479:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018051.270483:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332908928/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018051.270487:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018051.270489:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018051.270489:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018051.270491:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2313 00000100:00080000:0.0:1734018051.270493:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018051.270494:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018051.270496:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734018051.270498:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018051.270501:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332909056/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018051.270515:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018051/real 1734018051] req@ffff8800a705c380 x1818247332908288/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018051.270529:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332908288/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018051.270537:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018051.270540:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018051.270543:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332908288:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734018051.270552:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332908416:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734018051.270568:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018051.270575:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018051/real 1734018051] req@ffff8800a705d180 x1818247332908416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018051.270584:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332908416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018051.270591:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018051.270604:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018051.270607:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332908416:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734018051.270612:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332908544:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734018051.270626:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018051.270633:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018051/real 1734018051] req@ffff8800a705f100 x1818247332908544/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018051.270643:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332908544/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018051.270649:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018051.270651:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018051.270653:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332908544:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734018051.270675:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332908672:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734018051.270688:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018051.270695:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018051/real 1734018051] req@ffff8800a705dc00 x1818247332908672/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018051.270715:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332908672/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018051.270721:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018051.270724:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018051.270727:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332908672:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734018051.270731:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332908800:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734018051.270743:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018051.270748:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018051/real 1734018051] req@ffff8800a705df80 x1818247332908800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018051.270756:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332908800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018051.270763:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018051.270764:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018051.270768:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332908800:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734018051.270772:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332908928:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734018051.270784:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018051.270789:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018051/real 1734018051] req@ffff8800a705c700 x1818247332908928/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018051.270798:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332908928/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018051.270803:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018051.270806:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018051.270808:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332908928:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734018051.270813:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332909056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734018051.270824:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018051.270829:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018051/real 1734018051] req@ffff8800a705ed80 x1818247332909056/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018051.270837:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332909056/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018051.270844:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018051.270846:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734018051.270849:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332909056:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018051.810435:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018052.812563:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018053.814463:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018054.816431:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018055.818406:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018056.278359:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018056.278369:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018056.278373:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018056.278382:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2318 00000100:00080000:0.0:1734018056.278391:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018056.278397:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018056.278433:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332909184/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018056.278460:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018056.278462:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018056.278464:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018056.278467:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2318 00000100:00080000:0.0:1734018056.278469:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018056.278471:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018056.278479:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018056.278485:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332909312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018056.278491:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018056.278493:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018056.278494:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018056.278497:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2318 00000100:00080000:0.0:1734018056.278499:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018056.278501:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018056.278504:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018056.278509:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332909440/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018056.278515:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018056.278517:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018056.278518:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018056.278520:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2318 00000100:00080000:0.0:1734018056.278521:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018056.278523:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:2.0:1734018056.278524:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332909184:192.168.201.152@tcp:250:kworker.0 00000008:00100000:0.0:1734018056.278526:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018056.278531:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332909568/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018056.278535:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018056.278537:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018056.278538:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018056.278540:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2318 00000100:00080000:0.0:1734018056.278541:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018056.278543:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018056.278546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018056.278550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332909696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018056.278556:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018056.278557:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018056.278558:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018056.278561:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2318 00000100:00080000:0.0:1734018056.278563:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018056.278565:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018056.278568:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018056.278574:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332909824/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018056.278578:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018056.278579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018056.278580:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018056.278582:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2318 00000100:00080000:0.0:1734018056.278584:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018056.278585:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018056.278588:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018056.278592:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332909952/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734018056.278598:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332909312:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734018056.278617:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332909440:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734018056.278627:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332909568:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734018056.278635:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332909696:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734018056.278643:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332909824:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734018056.278652:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332909952:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018056.820371:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018057.822536:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018058.824394:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1734018059.284510:0:1957:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:1.0:1734018059.284556:0:1957:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1734018059.284573:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734018059.284603:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734018059.284608:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734018059.284613:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734018059.284618:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734018059.284623:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734018059.284627:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734018059.284668:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018056/real 1734018059] req@ffff8800a705e300 x1818247332909440/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018059.284688:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332909440/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018059.284704:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018059.284708:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018059.284712:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332909440:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018059.284719:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018056/real 1734018059] req@ffff8800a705c000 x1818247332909568/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018059.284728:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332909568/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018059.284735:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018059.284738:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018059.284741:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332909568:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018059.284747:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018056/real 1734018059] req@ffff8800a705e680 x1818247332909696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018059.284758:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332909696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018059.284765:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018059.284767:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018059.284770:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332909696:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018059.284776:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018056/real 1734018059] req@ffff8800a705ca80 x1818247332909824/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018059.284786:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332909824/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018059.284796:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018059.284799:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018059.284802:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332909824:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018059.284808:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018056/real 1734018059] req@ffff8800a705f800 x1818247332909952/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018059.284817:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332909952/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018059.284824:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018059.284827:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018059.284830:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332909952:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018059.284836:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018056/real 1734018059] req@ffff8800a705d880 x1818247332909184/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018059.284846:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332909184/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018059.284852:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018059.284854:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018059.284856:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332909184:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018059.284861:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018056/real 1734018059] req@ffff8800a705d500 x1818247332909312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018059.284871:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332909312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018059.284883:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018059.284886:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018059.284889:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332909312:192.168.201.152@tcp:38:kworker.0 00010000:00010000:2.0:1734018059.826410:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018060.828397:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018061.286331:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018061.286337:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018061.286340:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018061.286348:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2323 00000100:00080000:0.0:1734018061.286355:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018061.286360:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018061.286386:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332910080/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018061.286416:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018061.286418:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018061.286419:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018061.286422:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2323 00000100:00080000:0.0:1734018061.286423:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018061.286425:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018061.286431:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018061.286437:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332910208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018061.286442:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018061.286444:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018061.286445:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018061.286447:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2323 00000100:00080000:0.0:1734018061.286448:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018061.286450:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018061.286452:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018061.286458:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332910336/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018061.286462:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018061.286464:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018061.286465:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018061.286466:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2323 00000100:00080000:0.0:1734018061.286467:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018061.286469:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018061.286472:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018061.286476:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332910464/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018061.286480:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018061.286482:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018061.286482:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018061.286484:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2323 00000100:00080000:0.0:1734018061.286485:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018061.286487:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018061.286489:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018061.286493:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332910592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018061.286498:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018061.286498:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332910080:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018061.286500:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018061.286500:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018061.286502:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2323 00000100:00080000:0.0:1734018061.286503:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018061.286505:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018061.286507:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018061.286512:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332910720/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018061.286516:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018061.286518:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018061.286519:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018061.286521:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2323 00000100:00080000:0.0:1734018061.286522:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018061.286524:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018061.286526:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018061.286530:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332910848/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018061.286544:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018061.286558:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018061/real 1734018061] req@ffff8800a705ea00 x1818247332910080/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018061.286569:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332910080/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018061.286575:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018061.286576:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018061.286579:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332910080:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018061.286582:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332910208:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018061.286592:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018061.286596:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018061/real 1734018061] req@ffff8800a705fb80 x1818247332910208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018061.286601:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332910208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018061.286605:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018061.286606:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018061.286608:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332910208:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018061.286612:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332910336:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018061.286620:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018061.286623:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018061/real 1734018061] req@ffff8800a705d500 x1818247332910336/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018061.286628:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332910336/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018061.286632:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018061.286633:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018061.286635:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332910336:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018061.286637:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332910464:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018061.286643:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018061.286646:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018061/real 1734018061] req@ffff8800a705d880 x1818247332910464/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018061.286651:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332910464/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018061.286655:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018061.286656:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018061.286657:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332910464:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018061.286673:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332910592:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018061.286679:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018061.286682:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018061/real 1734018061] req@ffff8800a705f800 x1818247332910592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018061.286688:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332910592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018061.286692:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018061.286693:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018061.286694:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332910592:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018061.286697:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332910720:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018061.286703:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018061.286706:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018061/real 1734018061] req@ffff8800a705ca80 x1818247332910720/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018061.286711:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332910720/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018061.286714:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018061.286715:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018061.286717:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332910720:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018061.286720:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332910848:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018061.286725:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018061.286728:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018061/real 1734018061] req@ffff8800a705e680 x1818247332910848/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018061.286733:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332910848/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018061.286737:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018061.286739:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018061.286740:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332910848:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018061.830454:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018062.832532:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018063.834420:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018064.836418:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018065.838569:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018066.294372:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018066.294384:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018066.294390:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018066.294413:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2328 00000100:00080000:0.0:1734018066.294426:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018066.294434:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018066.294475:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332910976/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018066.294517:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018066.294522:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018066.294525:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018066.294531:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2328 00000100:00080000:0.0:1734018066.294535:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018066.294540:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018066.294552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018066.294567:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332911104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018066.294583:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018066.294587:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018066.294590:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018066.294594:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2328 00000100:00080000:0.0:1734018066.294598:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018066.294603:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018066.294625:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018066.294634:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332911232/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018066.294646:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018066.294647:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332910976:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018066.294650:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018066.294652:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018066.294656:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2328 00000100:00080000:0.0:1734018066.294659:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018066.294663:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018066.294669:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018066.294678:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332911360/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018066.294689:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018066.294693:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018066.294695:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018066.294700:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2328 00000100:00080000:0.0:1734018066.294704:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018066.294707:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018066.294708:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018066.294713:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018066.294723:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332911488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018066.294727:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018066/real 1734018066] req@ffff8800a705c000 x1818247332910976/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018066.294734:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018066.294737:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018066.294739:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018066.294742:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332910976/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018066.294744:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2328 00000100:00080000:0.0:1734018066.294747:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1734018066.294751:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018066.294752:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:1.0:1734018066.294754:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000008:00100000:0.0:1734018066.294757:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018066.294758:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332910976:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018066.294764:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332911104:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018066.294766:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332911616/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018066.294777:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734018066.294779:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018066.294781:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018066.294783:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018066.294785:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018066/real 1734018066] req@ffff8800a705e300 x1818247332911104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018066.294787:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2328 00000100:00080000:0.0:1734018066.294791:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018066.294794:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332911104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018066.294795:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018066.294800:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734018066.294800:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018066.294802:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018066.294805:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332911104:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018066.294811:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332911744/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018066.294825:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332911232:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018066.294839:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018066.294844:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018066/real 1734018066] req@ffff8800a705ed80 x1818247332911232/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018066.294853:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332911232/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018066.294859:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018066.294889:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018066.294892:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332911232:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018066.294897:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332911360:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018066.294909:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018066.294915:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018066/real 1734018066] req@ffff8800a705c700 x1818247332911360/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018066.294924:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332911360/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018066.294930:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018066.294932:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018066.294935:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332911360:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018066.294939:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332911488:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018066.294949:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018066.294954:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018066/real 1734018066] req@ffff8800a705df80 x1818247332911488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018066.294967:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332911488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018066.294973:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018066.294975:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018066.294977:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332911488:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018066.294981:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332911616:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018066.294991:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018066.294996:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018066/real 1734018066] req@ffff8800a705dc00 x1818247332911616/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018066.295003:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332911616/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018066.295009:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018066.295011:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018066.295013:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332911616:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018066.295026:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332911744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018066.295044:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018066.295049:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018066/real 1734018066] req@ffff8800a705f100 x1818247332911744/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018066.295057:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332911744/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018066.295063:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018066.295065:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018066.295067:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332911744:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018066.840523:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018067.842385:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018068.844420:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018069.846431:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018070.848511:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018071.302356:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018071.302369:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018071.302376:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018071.302390:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2333 00000100:00080000:0.0:1734018071.302403:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018071.302413:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018071.302463:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332911872/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018071.302502:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018071.302507:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018071.302509:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018071.302514:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2333 00000100:00080000:0.0:1734018071.302517:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018071.302522:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018071.302534:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018071.302546:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332912000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018071.302556:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018071.302559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018071.302561:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018071.302566:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2333 00000100:00080000:0.0:1734018071.302569:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018071.302572:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018071.302578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018071.302590:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332912128/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018071.302599:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018071.302602:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018071.302604:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018071.302607:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2333 00000100:00080000:0.0:1734018071.302610:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018071.302613:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018071.302618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018071.302626:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332912256/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018071.302636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018071.302639:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018071.302641:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018071.302643:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332911872:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018071.302645:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2333 00000100:00080000:0.0:1734018071.302647:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018071.302651:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018071.302656:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018071.302665:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332912384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018071.302675:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018071.302678:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018071.302680:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018071.302684:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2333 00000100:00080000:0.0:1734018071.302687:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018071.302691:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018071.302695:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018071.302703:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332912512/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018071.302711:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018071.302714:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018071.302716:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018071.302719:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2333 00000800:00000100:1.0:1734018071.302722:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018071.302723:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018071.302726:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018071.302730:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018071.302739:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332912640/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018071.302748:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018071/real 1734018071] req@ffff8800a705d180 x1818247332911872/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018071.302768:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332911872/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018071.302814:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018071.302818:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018071.302838:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332911872:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018071.302845:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332912000:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018071.302861:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018071.302866:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018071/real 1734018071] req@ffff8800a705c380 x1818247332912000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018071.302873:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332912000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018071.302878:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018071.302880:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018071.302882:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332912000:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018071.302886:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332912128:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018071.302897:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018071.302901:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018071/real 1734018071] req@ffff8800a705f100 x1818247332912128/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018071.302908:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332912128/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018071.302912:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018071.302913:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018071.302915:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332912128:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018071.302934:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332912256:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018071.302941:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018071.302945:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018071/real 1734018071] req@ffff8800a705dc00 x1818247332912256/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018071.302951:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332912256/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018071.302955:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018071.302957:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018071.302958:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332912256:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018071.302962:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332912384:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018071.302970:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018071.302974:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018071/real 1734018071] req@ffff8800a705df80 x1818247332912384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018071.302984:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332912384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018071.302988:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018071.302990:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018071.302991:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332912384:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018071.302995:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332912512:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018071.303002:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018071.303006:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018071/real 1734018071] req@ffff8800a705c700 x1818247332912512/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018071.303011:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332912512/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018071.303015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018071.303017:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018071.303018:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332912512:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018071.303022:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332912640:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018071.303030:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018071.303034:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018071/real 1734018071] req@ffff8800a705ed80 x1818247332912640/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018071.303040:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332912640/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018071.303044:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018071.303045:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018071.303047:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332912640:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018071.850522:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018072.852530:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018073.854474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018074.856480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018075.858488:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018076.310328:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018076.310337:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018076.310342:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018076.310353:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2338 00000100:00080000:0.0:1734018076.310363:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018076.310370:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018076.310404:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332912768/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018076.310445:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018076.310449:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018076.310451:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018076.310455:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2338 00000100:00080000:0.0:1734018076.310459:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018076.310463:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018076.310472:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018076.310483:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332912896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018076.310492:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018076.310496:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018076.310497:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018076.310501:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2338 00000100:00080000:0.0:1734018076.310504:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018076.310507:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018076.310512:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018076.310520:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332913024/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018076.310528:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018076.310531:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018076.310532:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018076.310536:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2338 00000100:00080000:0.0:1734018076.310538:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018076.310542:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018076.310546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018076.310555:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332913152/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018076.310560:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332912768:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018076.310563:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018076.310567:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018076.310568:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018076.310571:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2338 00000100:00080000:0.0:1734018076.310574:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018076.310577:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018076.310581:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018076.310589:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332913280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018076.310597:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018076.310599:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018076.310601:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018076.310605:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2338 00000100:00080000:0.0:1734018076.310616:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018076.310620:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018076.310624:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018076.310626:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018076.310634:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332913408/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018076.310643:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018076.310646:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018076.310648:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018076.310648:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018076/real 1734018076] req@ffff8800a705e300 x1818247332912768/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018076.310651:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2338 00000100:00080000:0.0:1734018076.310655:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018076.310658:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018076.310662:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018076.310667:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332912768/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018076.310670:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332913536/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018076.310678:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018076.310681:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018076.310686:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332912768:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018076.310722:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332912896:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018076.310743:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018076.310751:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018076/real 1734018076] req@ffff8800a705c000 x1818247332912896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018076.310763:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332912896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018076.310771:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018076.310774:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018076.310777:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332912896:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018076.310782:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332913024:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018076.310799:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018076.310805:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018076/real 1734018076] req@ffff8800a705e680 x1818247332913024/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018076.310816:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332913024/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018076.310824:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018076.310826:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018076.310829:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332913024:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018076.310853:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332913152:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018076.310865:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018076.310871:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018076/real 1734018076] req@ffff8800a705ca80 x1818247332913152/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018076.310881:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332913152/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018076.310888:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018076.310891:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018076.310894:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332913152:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018076.310899:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332913280:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018076.310911:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018076.310917:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018076/real 1734018076] req@ffff8800a705f800 x1818247332913280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018076.310934:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332913280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018076.310941:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018076.310943:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018076.310946:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332913280:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018076.310952:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332913408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018076.310963:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018076.310969:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018076/real 1734018076] req@ffff8800a705d880 x1818247332913408/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018076.310979:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332913408/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018076.310986:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018076.310988:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018076.310991:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332913408:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018076.310996:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332913536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018076.311008:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018076.311014:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018076/real 1734018076] req@ffff8800a705d500 x1818247332913536/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018076.311023:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332913536/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018076.311031:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018076.311033:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018076.311036:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332913536:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018076.860517:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018077.862507:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018078.864521:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018079.866459:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018080.868480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018081.318360:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018081.318369:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018081.318374:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018081.318385:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2343 00000100:00080000:0.0:1734018081.318395:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018081.318403:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018081.318440:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332913664/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018081.318473:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018081.318477:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018081.318480:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018081.318484:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2343 00000100:00080000:0.0:1734018081.318488:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018081.318491:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018081.318502:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018081.318512:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332913792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018081.318523:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018081.318526:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018081.318528:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018081.318531:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2343 00000100:00080000:0.0:1734018081.318534:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018081.318538:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018081.318542:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018081.318552:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332913920/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018081.318560:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018081.318563:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018081.318565:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018081.318568:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2343 00000100:00080000:0.0:1734018081.318571:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018081.318574:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018081.318578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018081.318586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332914048/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018081.318588:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332913664:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018081.318595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018081.318599:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018081.318600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018081.318603:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2343 00000100:00080000:0.0:1734018081.318644:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018081.318647:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018081.318652:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018081.318661:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332914176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018081.318670:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018081.318674:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018081.318675:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018081.318679:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2343 00000100:00080000:0.0:1734018081.318682:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018081.318685:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018081.318687:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734018081.318689:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018081.318697:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332914304/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018081.318705:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018081.318708:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734018081.318709:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018081/real 1734018081] req@ffff8800a705fb80 x1818247332913664/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018081.318710:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018081.318713:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2343 00000100:00080000:0.0:1734018081.318716:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018081.318719:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018081.318723:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018081.318728:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332913664/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018081.318731:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332914432/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018081.318739:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018081.318742:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018081.318747:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332913664:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018081.318754:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332913792:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018081.318773:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018081.318780:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018081/real 1734018081] req@ffff8800a705ea00 x1818247332913792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018081.318791:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332913792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018081.318799:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018081.318801:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018081.318805:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332913792:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018081.318810:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332913920:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018081.318827:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018081.318833:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018081/real 1734018081] req@ffff8800a705d500 x1818247332913920/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018081.318843:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332913920/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018081.318851:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018081.318854:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018081.318856:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332913920:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018081.318880:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332914048:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018081.318892:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018081.318898:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018081/real 1734018081] req@ffff8800a705d880 x1818247332914048/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018081.318908:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332914048/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018081.318916:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018081.318918:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018081.318921:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332914048:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018081.318927:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332914176:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018081.318938:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018081.318944:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018081/real 1734018081] req@ffff8800a705f800 x1818247332914176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018081.318961:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332914176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018081.318968:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018081.318971:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018081.318973:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332914176:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018081.318979:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332914304:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018081.318990:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018081.318996:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018081/real 1734018081] req@ffff8800a705ca80 x1818247332914304/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018081.319006:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332914304/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018081.319013:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018081.319015:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018081.319018:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332914304:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018081.319024:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332914432:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018081.319035:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018081.319041:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018081/real 1734018081] req@ffff8800a705e680 x1818247332914432/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018081.319051:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332914432/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018081.319058:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018081.319061:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018081.319064:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332914432:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018081.870440:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018082.872452:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018083.874518:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018084.876507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018085.878471:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018086.326326:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018086.326335:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018086.326340:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018086.326349:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2348 00000100:00080000:0.0:1734018086.326358:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018086.326367:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018086.326402:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332914560/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018086.326435:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018086.326439:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018086.326441:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018086.326446:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2348 00000100:00080000:0.0:1734018086.326449:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018086.326453:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018086.326463:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018086.326474:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332914688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018086.326484:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018086.326487:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018086.326489:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018086.326493:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2348 00000100:00080000:0.0:1734018086.326495:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018086.326499:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018086.326504:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018086.326513:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332914816/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018086.326555:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018086.326558:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018086.326560:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018086.326564:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2348 00000100:00080000:0.0:1734018086.326567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018086.326570:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018086.326575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018086.326584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332914944/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018086.326585:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332914560:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018086.326593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018086.326596:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018086.326598:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018086.326601:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2348 00000100:00080000:0.0:1734018086.326605:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018086.326609:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018086.326613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018086.326621:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332915072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018086.326629:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018086.326632:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018086.326634:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018086.326638:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2348 00000100:00080000:0.0:1734018086.326641:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018086.326644:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018086.326648:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018086.326651:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018086.326656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332915200/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018086.326664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018086.326667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018086.326668:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018086.326672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2348 00000100:00080000:0.0:1734018086.326674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018086.326674:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018086/real 1734018086] req@ffff8800a705c000 x1818247332914560/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018086.326678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018086.326682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018086.326691:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332915328/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018086.326693:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332914560/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018086.326705:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018086.326708:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018086.326712:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332914560:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018086.326720:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332914688:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018086.326737:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018086.326744:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018086/real 1734018086] req@ffff8800a705e300 x1818247332914688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018086.326755:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332914688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018086.326763:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018086.326765:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018086.326768:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332914688:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018086.326774:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332914816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018086.326791:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018086.326797:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018086/real 1734018086] req@ffff8800a705ed80 x1818247332914816/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018086.326808:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332914816/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018086.326815:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018086.326817:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018086.326820:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332914816:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018086.326845:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332914944:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018086.326857:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018086.326863:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018086/real 1734018086] req@ffff8800a705c700 x1818247332914944/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018086.326873:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332914944/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018086.326880:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018086.326883:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018086.326886:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332914944:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018086.326891:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332915072:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018086.326903:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018086.326908:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018086/real 1734018086] req@ffff8800a705df80 x1818247332915072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018086.326925:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332915072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018086.326932:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018086.326934:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018086.326938:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332915072:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018086.326943:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332915200:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018086.326954:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018086.326960:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018086/real 1734018086] req@ffff8800a705dc00 x1818247332915200/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018086.326970:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332915200/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018086.326977:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018086.326979:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018086.326982:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332915200:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018086.326987:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332915328:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018086.326999:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018086.327005:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018086/real 1734018086] req@ffff8800a705f100 x1818247332915328/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018086.327015:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332915328/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018086.327022:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018086.327024:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018086.327027:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332915328:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018086.880559:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018087.882480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018088.884569:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018089.886545:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018090.888535:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018091.334369:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018091.334378:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018091.334382:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018091.334393:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2353 00000100:00080000:0.0:1734018091.334403:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018091.334411:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018091.334492:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332915456/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018091.334532:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018091.334536:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018091.334538:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018091.334543:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2353 00000100:00080000:0.0:1734018091.334547:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018091.334551:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018091.334561:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018091.334573:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332915584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018091.334583:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018091.334586:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018091.334588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018091.334591:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2353 00000100:00080000:0.0:1734018091.334594:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018091.334597:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018091.334602:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018091.334612:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332915712/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018091.334620:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018091.334623:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018091.334625:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018091.334628:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2353 00000100:00080000:0.0:1734018091.334631:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018091.334634:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018091.334639:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018091.334647:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332915840/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018091.334647:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332915456:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018091.334656:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018091.334659:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018091.334661:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018091.334664:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2353 00000100:00080000:0.0:1734018091.334668:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018091.334671:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018091.334675:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018091.334683:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332915968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018091.334691:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018091.334694:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018091.334696:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018091.334699:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2353 00000100:00080000:0.0:1734018091.334702:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018091.334705:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018091.334710:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018091.334714:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018091.334717:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332916096/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018091.334725:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018091.334729:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018091.334730:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018091.334734:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2353 00000100:00080000:0.0:1734018091.334736:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018091.334737:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018091/real 1734018091] req@ffff8800a705c380 x1818247332915456/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018091.334740:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018091.334744:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018091.334752:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332916224/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018091.334757:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332915456/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018091.334767:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018091.334770:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018091.334775:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332915456:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018091.334782:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332915584:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018091.334799:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018091.334806:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018091/real 1734018091] req@ffff8800a705d180 x1818247332915584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018091.334817:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332915584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018091.334825:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018091.334828:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018091.334831:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332915584:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018091.334837:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332915712:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018091.334853:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018091.334859:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018091/real 1734018091] req@ffff8800a705f100 x1818247332915712/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018091.334869:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332915712/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018091.334877:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018091.334879:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018091.334882:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332915712:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018091.334907:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332915840:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018091.334919:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018091.334925:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018091/real 1734018091] req@ffff8800a705dc00 x1818247332915840/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018091.334935:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332915840/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018091.334942:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018091.334944:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018091.334948:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332915840:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018091.334953:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332915968:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018091.334965:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018091.334971:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018091/real 1734018091] req@ffff8800a705df80 x1818247332915968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018091.334987:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332915968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018091.334995:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018091.334997:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018091.335000:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332915968:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018091.335006:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332916096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018091.335017:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018091.335023:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018091/real 1734018091] req@ffff8800a705c700 x1818247332916096/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018091.335033:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332916096/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018091.335040:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018091.335042:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018091.335045:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332916096:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018091.335050:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332916224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018091.335062:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018091.335068:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018091/real 1734018091] req@ffff8800a705ed80 x1818247332916224/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018091.335078:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332916224/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018091.335085:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018091.335088:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018091.335091:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332916224:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018091.890506:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018092.892491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018093.894470:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018094.896470:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018095.898551:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018096.342273:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018096.342279:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018096.342282:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018096.342288:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2358 00000100:00080000:0.0:1734018096.342295:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018096.342300:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018096.342322:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332916352/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018096.342344:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018096.342346:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018096.342347:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018096.342361:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2358 00000100:00080000:0.0:1734018096.342363:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018096.342365:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018096.342372:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018096.342378:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332916480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018096.342387:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018096.342389:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018096.342390:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018096.342392:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2358 00000100:00080000:0.0:1734018096.342394:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018096.342396:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018096.342399:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018096.342403:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332916608/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018096.342407:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018096.342409:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018096.342410:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018096.342412:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2358 00000100:00080000:0.0:1734018096.342413:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018096.342415:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018096.342417:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018096.342421:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332916736/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018096.342425:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332916352:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018096.342426:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018096.342427:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018096.342428:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018096.342430:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2358 00000100:00080000:0.0:1734018096.342432:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018096.342434:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018096.342436:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018096.342440:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332916864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018096.342444:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018096.342446:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018096.342447:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018096.342449:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2358 00000100:00080000:0.0:1734018096.342450:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018096.342452:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018096.342454:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018096.342459:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332916992/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018096.342463:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018096.342465:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018096.342465:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018096.342467:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2358 00000100:00080000:0.0:1734018096.342468:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018096.342469:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018096.342470:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018096.342472:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018096.342477:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332917120/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018096.342483:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018096/real 1734018096] req@ffff8800a705e300 x1818247332916352/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018096.342494:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332916352/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018096.342500:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018096.342502:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018096.342504:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332916352:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018096.342508:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332916480:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018096.342518:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018096.342521:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018096/real 1734018096] req@ffff8800a705c000 x1818247332916480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018096.342527:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332916480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018096.342531:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018096.342532:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018096.342534:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332916480:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018096.342537:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332916608:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018096.342546:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018096.342549:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018096/real 1734018096] req@ffff8800a705e680 x1818247332916608/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018096.342554:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332916608/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018096.342558:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018096.342559:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018096.342561:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332916608:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018096.342575:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332916736:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018096.342581:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018096.342584:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018096/real 1734018096] req@ffff8800a705ca80 x1818247332916736/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018096.342589:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332916736/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018096.342593:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018096.342594:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018096.342596:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332916736:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018096.342599:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332916864:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018096.342605:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018096.342608:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018096/real 1734018096] req@ffff8800a705f800 x1818247332916864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018096.342617:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332916864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018096.342621:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018096.342622:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018096.342623:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332916864:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018096.342626:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332916992:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018096.342632:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018096.342635:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018096/real 1734018096] req@ffff8800a705d880 x1818247332916992/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018096.342640:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332916992/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018096.342643:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018096.342644:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018096.342646:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332916992:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018096.342649:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332917120:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018096.342655:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018096.342658:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018096/real 1734018096] req@ffff8800a705d500 x1818247332917120/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018096.342663:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332917120/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018096.342667:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018096.342668:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018096.342670:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332917120:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018096.900529:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018097.902550:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018098.904560:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018099.906481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018100.908514:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018101.350360:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018101.350372:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018101.350378:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018101.350392:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2363 00000100:00080000:0.0:1734018101.350404:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018101.350414:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018101.350461:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332917248/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018101.350497:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018101.350502:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018101.350504:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018101.350509:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2363 00000100:00080000:0.0:1734018101.350512:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018101.350517:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018101.350528:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018101.350540:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332917376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018101.350550:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018101.350554:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018101.350555:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018101.350560:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2363 00000100:00080000:0.0:1734018101.350562:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018101.350566:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018101.350572:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018101.350584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332917504/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018101.350594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018101.350597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018101.350598:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018101.350602:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2363 00000100:00080000:0.0:1734018101.350604:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018101.350608:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018101.350612:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018101.350620:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332917632/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018101.350629:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018101.350633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018101.350635:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018101.350639:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2363 00000100:00080000:0.0:1734018101.350642:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018101.350644:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332917248:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018101.350646:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018101.350650:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018101.350659:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332917760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018101.350667:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018101.350670:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018101.350672:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018101.350677:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2363 00000100:00080000:0.0:1734018101.350680:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018101.350683:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018101.350688:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018101.350696:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332917888/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018101.350705:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018101.350708:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018101.350709:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018101.350713:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2363 00000100:00080000:0.0:1734018101.350716:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018101.350720:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018101.350720:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734018101.350724:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018101.350733:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332918016/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018101.350744:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018101/real 1734018101] req@ffff8800a705ea00 x1818247332917248/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018101.350765:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332917248/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018101.350777:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018101.350780:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018101.350785:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332917248:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018101.350793:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332917376:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018101.350811:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018101.350818:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018101/real 1734018101] req@ffff8800a705fb80 x1818247332917376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018101.350829:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332917376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018101.350837:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018101.350840:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018101.350843:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332917376:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018101.350849:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332917504:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018101.350867:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018101.350873:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018101/real 1734018101] req@ffff8800a705d500 x1818247332917504/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018101.350884:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332917504/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018101.350891:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018101.350893:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018101.350896:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332917504:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018101.350924:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332917632:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018101.350937:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018101.350943:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018101/real 1734018101] req@ffff8800a705d880 x1818247332917632/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018101.350953:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332917632/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018101.350961:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018101.350963:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018101.350966:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332917632:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018101.350973:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332917760:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018101.350985:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018101.350991:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018101/real 1734018101] req@ffff8800a705f800 x1818247332917760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018101.351008:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332917760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018101.351016:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018101.351018:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018101.351021:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332917760:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018101.351027:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332917888:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018101.351039:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018101.351045:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018101/real 1734018101] req@ffff8800a705ca80 x1818247332917888/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018101.351055:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332917888/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018101.351062:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018101.351064:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018101.351067:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332917888:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018101.351072:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332918016:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018101.351085:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018101.351091:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018101/real 1734018101] req@ffff8800a705e680 x1818247332918016/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018101.351101:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332918016/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018101.351108:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018101.351111:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018101.351114:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332918016:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018101.910515:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018102.912525:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018103.914483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018104.916520:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018105.918448:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018106.358360:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018106.358369:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018106.358373:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018106.358384:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2368 00000100:00080000:0.0:1734018106.358394:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018106.358402:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018106.358436:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332918144/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018106.358478:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018106.358482:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018106.358484:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018106.358488:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2368 00000100:00080000:0.0:1734018106.358492:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018106.358495:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018106.358505:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018106.358516:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332918272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018106.358526:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018106.358530:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018106.358531:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018106.358535:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2368 00000100:00080000:0.0:1734018106.358537:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018106.358540:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018106.358545:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018106.358553:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332918400/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018106.358562:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018106.358565:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018106.358566:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018106.358570:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2368 00000100:00080000:0.0:1734018106.358572:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018106.358576:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018106.358580:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018106.358588:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332918528/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018106.358594:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332918144:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018106.358596:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018106.358599:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018106.358601:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018106.358604:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2368 00000100:00080000:0.0:1734018106.358607:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018106.358610:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018106.358614:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018106.358622:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332918656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018106.358630:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018106.358633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018106.358634:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018106.358638:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2368 00000100:00080000:0.0:1734018106.358649:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018106.358653:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018106.358658:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018106.358662:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018106.358667:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332918784/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018106.358676:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018106.358680:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018106.358681:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018106.358684:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018106/real 1734018106] req@ffff8800a705c000 x1818247332918144/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018106.358685:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2368 00000100:00080000:0.0:1734018106.358688:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018106.358691:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018106.358695:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018106.358702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332918912/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018106.358704:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332918144/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018106.358715:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018106.358718:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018106.358723:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332918144:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018106.358730:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332918272:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018106.358748:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018106.358754:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018106/real 1734018106] req@ffff8800a705e300 x1818247332918272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018106.358765:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332918272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018106.358773:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018106.358776:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018106.358779:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332918272:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018106.358785:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332918400:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018106.358802:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018106.358808:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018106/real 1734018106] req@ffff8800a705ed80 x1818247332918400/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018106.358819:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332918400/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018106.358826:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018106.358828:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018106.358831:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332918400:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018106.358856:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332918528:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018106.358868:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018106.358874:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018106/real 1734018106] req@ffff8800a705c700 x1818247332918528/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018106.358884:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332918528/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018106.358891:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018106.358894:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018106.358897:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332918528:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018106.358903:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332918656:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018106.358915:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018106.358921:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018106/real 1734018106] req@ffff8800a705df80 x1818247332918656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018106.358937:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332918656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018106.358945:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018106.358947:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018106.358950:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332918656:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018106.358955:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332918784:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018106.358967:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018106.358972:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018106/real 1734018106] req@ffff8800a705dc00 x1818247332918784/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018106.358982:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332918784/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018106.358989:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018106.358991:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018106.358994:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332918784:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018106.359000:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332918912:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018106.359012:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018106.359017:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018106/real 1734018106] req@ffff8800a705f100 x1818247332918912/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018106.359027:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332918912/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018106.359034:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018106.359037:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018106.359040:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332918912:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018106.920466:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018107.922494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018108.924526:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018109.926505:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018110.928509:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018111.366374:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018111.366385:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018111.366391:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018111.366405:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2373 00000100:00080000:0.0:1734018111.366416:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018111.366426:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018111.366474:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332919040/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018111.366513:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018111.366518:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018111.366520:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018111.366525:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2373 00000100:00080000:0.0:1734018111.366529:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018111.366533:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018111.366545:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018111.366557:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332919168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018111.366568:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018111.366572:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018111.366574:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018111.366577:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2373 00000100:00080000:0.0:1734018111.366580:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018111.366584:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018111.366589:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018111.366599:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332919296/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018111.366609:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018111.366612:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018111.366613:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018111.366617:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2373 00000100:00080000:0.0:1734018111.366620:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018111.366623:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018111.366628:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018111.366637:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332919424/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018111.366646:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018111.366650:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018111.366652:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018111.366653:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332919040:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018111.366656:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2373 00000100:00080000:0.0:1734018111.366659:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018111.366663:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018111.366668:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018111.366677:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332919552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018111.366685:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018111.366689:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018111.366690:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018111.366694:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2373 00000100:00080000:0.0:1734018111.366697:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018111.366700:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018111.366704:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018111.366712:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332919680/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018111.366720:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018111.366723:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018111.366725:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018111.366728:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2373 00000100:00080000:0.0:1734018111.366731:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018111.366732:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018111.366735:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018111.366739:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018111.366748:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332919808/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018111.366756:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018111/real 1734018111] req@ffff8800a705d180 x1818247332919040/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018111.366776:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332919040/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018111.366788:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018111.366792:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018111.366797:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332919040:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018111.366804:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332919168:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018111.366822:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018111.366829:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018111/real 1734018111] req@ffff8800a705c380 x1818247332919168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018111.366840:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332919168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018111.366848:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018111.366851:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018111.366854:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332919168:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018111.366860:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332919296:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018111.366878:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018111.366884:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018111/real 1734018111] req@ffff8800a705f100 x1818247332919296/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018111.366895:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332919296/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018111.366902:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018111.366905:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018111.366907:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332919296:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018111.366933:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332919424:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018111.366946:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018111.366952:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018111/real 1734018111] req@ffff8800a705dc00 x1818247332919424/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018111.366962:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332919424/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018111.366969:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018111.366972:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018111.366975:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332919424:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018111.366981:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332919552:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018111.366994:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018111.366999:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018111/real 1734018111] req@ffff8800a705df80 x1818247332919552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018111.367016:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332919552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018111.367024:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018111.367026:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018111.367029:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332919552:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018111.367035:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332919680:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018111.367047:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018111.367053:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018111/real 1734018111] req@ffff8800a705c700 x1818247332919680/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018111.367063:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332919680/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018111.367070:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018111.367072:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018111.367075:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332919680:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018111.367081:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332919808:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018111.367143:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018111.367152:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018111/real 1734018111] req@ffff8800a705ed80 x1818247332919808/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018111.367163:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332919808/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018111.367171:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018111.367175:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018111.367178:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332919808:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018111.930524:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018112.932521:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018113.934498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018114.936514:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018115.938514:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018116.374355:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018116.374367:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018116.374372:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018116.374386:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2378 00000100:00080000:0.0:1734018116.374397:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018116.374407:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018116.374452:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332919936/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018116.374489:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018116.374494:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018116.374496:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018116.374501:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2378 00000100:00080000:0.0:1734018116.374504:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018116.374509:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018116.374521:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018116.374533:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332920064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018116.374544:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018116.374547:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018116.374549:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018116.374553:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2378 00000100:00080000:0.0:1734018116.374556:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018116.374560:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018116.374566:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018116.374575:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332920192/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018116.374584:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018116.374587:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018116.374589:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018116.374593:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2378 00000100:00080000:0.0:1734018116.374595:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018116.374599:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018116.374603:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018116.374612:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332920320/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018116.374621:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018116.374623:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332919936:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018116.374624:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018116.374626:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018116.374630:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2378 00000100:00080000:0.0:1734018116.374633:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018116.374636:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018116.374641:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018116.374650:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332920448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018116.374659:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018116.374662:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018116.374664:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018116.374669:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2378 00000100:00080000:0.0:1734018116.374683:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018116.374687:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018116.374693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018116.374697:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018116.374703:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332920576/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018116.374712:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018116.374715:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018116.374717:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018116.374721:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2378 00000100:00100000:1.0:1734018116.374721:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018116/real 1734018116] req@ffff8800a705e300 x1818247332919936/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018116.374724:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018116.374728:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018116.374733:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018116.374742:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332920704/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018116.374742:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332919936/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018116.374754:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018116.374758:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018116.374763:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332919936:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018116.374771:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332920064:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018116.374790:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018116.374796:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018116/real 1734018116] req@ffff8800a705c000 x1818247332920064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018116.374807:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332920064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018116.374816:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018116.374818:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018116.374822:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332920064:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018116.374828:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332920192:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018116.374845:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018116.374852:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018116/real 1734018116] req@ffff8800a705e680 x1818247332920192/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018116.374862:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332920192/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018116.374870:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018116.374872:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018116.374876:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332920192:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018116.374901:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332920320:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018116.374913:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018116.374920:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018116/real 1734018116] req@ffff8800a705ca80 x1818247332920320/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018116.374930:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332920320/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018116.374937:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018116.374940:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018116.374943:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332920320:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018116.374949:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332920448:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018116.374961:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018116.374967:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018116/real 1734018116] req@ffff8800a705f800 x1818247332920448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018116.374984:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332920448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018116.374992:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018116.374994:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018116.374997:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332920448:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018116.375003:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332920576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018116.375050:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018116.375057:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018116/real 1734018116] req@ffff8800a705d880 x1818247332920576/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018116.375067:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332920576/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018116.375075:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018116.375077:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018116.375080:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332920576:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018116.375086:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332920704:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018116.375099:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018116.375104:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018116/real 1734018116] req@ffff8800a705d500 x1818247332920704/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018116.375115:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332920704/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018116.375122:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018116.375125:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018116.375128:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332920704:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018116.940494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018117.942481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018118.944495:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018119.946460:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018120.948445:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018121.382372:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018121.382384:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018121.382390:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018121.382405:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2383 00000100:00080000:0.0:1734018121.382416:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018121.382426:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018121.382476:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332920832/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018121.382520:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018121.382525:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018121.382527:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018121.382532:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2383 00000100:00080000:0.0:1734018121.382536:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018121.382540:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018121.382553:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018121.382565:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332920960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018121.382576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018121.382579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018121.382581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018121.382585:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2383 00000100:00080000:0.0:1734018121.382588:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018121.382591:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018121.382596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018121.382607:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332921088/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018121.382615:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018121.382619:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018121.382620:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018121.382624:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2383 00000100:00080000:0.0:1734018121.382627:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018121.382631:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018121.382635:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332920832:192.168.201.152@tcp:250:kworker.0 00000008:00100000:0.0:1734018121.382636:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018121.382644:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332921216/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018121.382654:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018121.382657:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018121.382659:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018121.382662:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2383 00000100:00080000:0.0:1734018121.382665:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018121.382668:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018121.382673:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018121.382681:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332921344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018121.382689:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018121.382692:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018121.382694:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018121.382698:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2383 00000100:00080000:0.0:1734018121.382702:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018121.382706:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018121.382710:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018121.382718:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332921472/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018121.382727:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018121.382731:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018121.382733:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018121.382737:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2383 00000100:00080000:0.0:1734018121.382740:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018121.382743:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018121.382748:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018121.382749:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332920960:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018121.382756:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332921600/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018121.382772:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332921088:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018121.382799:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332921216:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018121.382816:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332921344:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018121.382831:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332921472:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018121.382847:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332921600:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018121.950510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018122.952533:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018123.954514:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734018124.388674:0:1958:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734018124.388751:0:1958:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734018124.388775:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018124.388822:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018124.388831:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018124.388841:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018124.388848:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018124.388855:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018124.388862:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734018124.388962:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018121/real 1734018124] req@ffff8800a705d880 x1818247332921216/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018124.388997:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332921216/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018124.389019:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018124.389024:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018124.389030:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332921216:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018124.389039:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018121/real 1734018124] req@ffff8800a705f800 x1818247332921344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018124.389052:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332921344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018124.389063:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018124.389068:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018124.389073:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332921344:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018124.389084:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018121/real 1734018124] req@ffff8800a705ca80 x1818247332921472/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018124.389104:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332921472/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018124.389116:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018124.389121:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018124.389126:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332921472:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018124.389137:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018121/real 1734018124] req@ffff8800a705e680 x1818247332921600/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018124.389154:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332921600/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018124.389166:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018124.389171:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018124.389177:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332921600:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018124.389187:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018121/real 1734018124] req@ffff8800a705fb80 x1818247332920832/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018124.389206:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332920832/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018124.389401:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018124.389406:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018124.389413:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332920832:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018124.389425:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018121/real 1734018124] req@ffff8800a705ea00 x1818247332920960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018124.389455:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332920960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018124.389467:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018124.389472:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018124.389478:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332920960:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018124.389489:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018121/real 1734018124] req@ffff8800a705d500 x1818247332921088/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018124.389507:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332921088/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018124.389529:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018124.389533:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018124.389539:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332921088:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018124.956468:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018125.958463:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018126.390331:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018126.390341:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018126.390345:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018126.390357:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2388 00000100:00080000:0.0:1734018126.390366:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018126.390374:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018126.390410:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332921728/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018126.390453:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018126.390457:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018126.390459:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018126.390464:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2388 00000100:00080000:0.0:1734018126.390467:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018126.390471:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018126.390481:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018126.390492:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332921856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018126.390501:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018126.390504:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018126.390506:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018126.390510:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2388 00000100:00080000:0.0:1734018126.390513:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018126.390516:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018126.390521:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018126.390529:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332921984/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018126.390537:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018126.390540:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018126.390542:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018126.390545:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2388 00000100:00080000:0.0:1734018126.390548:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018126.390552:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018126.390557:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018126.390564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332922112/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018126.390572:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018126.390576:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018126.390577:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018126.390580:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2388 00000100:00080000:0.0:1734018126.390583:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018126.390586:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018126.390591:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018126.390598:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332922240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018126.390608:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018126.390610:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018126.390612:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018126.390613:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332921728:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018126.390616:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2388 00000100:00080000:0.0:1734018126.390618:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018126.390622:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018126.390626:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018126.390636:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332922368/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018126.390645:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018126.390648:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018126.390650:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018126.390654:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2388 00000100:00080000:0.0:1734018126.390656:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018126.390660:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018126.390664:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018126.390671:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332922496/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018126.390708:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018126.390737:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018126/real 1734018126] req@ffff8800a705c000 x1818247332921728/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018126.390767:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332921728/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018126.390780:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018126.390784:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018126.390789:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332921728:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018126.390797:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332921856:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018126.390817:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018126.390823:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018126/real 1734018126] req@ffff8800a705e300 x1818247332921856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018126.390834:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332921856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018126.390896:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018126.390900:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018126.390904:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332921856:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018126.390912:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332921984:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018126.390931:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018126.390939:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018126/real 1734018126] req@ffff8800a705ed80 x1818247332921984/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018126.390951:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332921984/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018126.390959:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018126.390962:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018126.390965:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332921984:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018126.390972:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332922112:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018126.390985:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018126.390991:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018126/real 1734018126] req@ffff8800a705c700 x1818247332922112/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018126.391001:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332922112/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018126.391010:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018126.391012:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018126.391015:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332922112:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018126.391047:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332922240:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018126.391060:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018126.391066:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018126/real 1734018126] req@ffff8800a705df80 x1818247332922240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018126.391076:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332922240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018126.391085:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018126.391087:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018126.391090:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332922240:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018126.391096:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332922368:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018126.391113:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018126.391119:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018126/real 1734018126] req@ffff8800a705dc00 x1818247332922368/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018126.391130:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332922368/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018126.391137:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018126.391139:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018126.391142:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332922368:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018126.391148:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332922496:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018126.391160:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018126.391166:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018126/real 1734018126] req@ffff8800a705f100 x1818247332922496/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018126.391176:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332922496/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018126.391183:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018126.391185:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018126.391188:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332922496:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018126.960514:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018127.962501:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018128.964511:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018129.966474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018130.968482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018131.398323:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018131.398332:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018131.398336:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018131.398347:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2393 00000100:00080000:0.0:1734018131.398357:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018131.398364:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018131.398399:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332922624/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018131.398433:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018131.398437:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018131.398439:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018131.398444:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2393 00000100:00080000:0.0:1734018131.398447:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018131.398451:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018131.398461:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018131.398472:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332922752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018131.398483:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018131.398486:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018131.398488:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018131.398491:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2393 00000100:00080000:0.0:1734018131.398494:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018131.398497:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018131.398502:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018131.398512:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332922880/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018131.398520:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018131.398524:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018131.398525:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018131.398529:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2393 00000100:00080000:0.0:1734018131.398531:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018131.398534:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018131.398539:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018131.398547:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332923008/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018131.398558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018131.398561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018131.398563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018131.398566:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2393 00000100:00080000:0.0:1734018131.398569:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018131.398572:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018131.398577:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734018131.398583:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332922624:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018131.398585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332923136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018131.398593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018131.398596:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018131.398598:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018131.398601:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2393 00000100:00080000:0.0:1734018131.398603:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018131.398607:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018131.398611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018131.398618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332923264/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018131.398626:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018131.398629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018131.398630:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018131.398634:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2393 00000100:00080000:0.0:1734018131.398636:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018131.398639:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018131.398643:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018131.398651:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332923392/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018131.398680:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018131.398711:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018131/real 1734018131] req@ffff8800a705c380 x1818247332922624/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018131.398742:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332922624/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018131.398800:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018131.398805:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018131.398811:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332922624:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018131.398820:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332922752:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018131.398843:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018131.398852:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018131/real 1734018131] req@ffff8800a705d180 x1818247332922752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018131.398863:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332922752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018131.398871:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018131.398875:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018131.398878:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332922752:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018131.398885:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332922880:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018131.398900:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018131.398906:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018131/real 1734018131] req@ffff8800a705f100 x1818247332922880/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018131.398916:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332922880/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018131.398925:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018131.398928:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018131.398932:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332922880:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018131.398963:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332923008:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018131.398977:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018131.398983:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018131/real 1734018131] req@ffff8800a705dc00 x1818247332923008/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018131.398994:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332923008/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018131.399002:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018131.399004:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018131.399007:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332923008:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018131.399013:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332923136:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018131.399026:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018131.399031:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018131/real 1734018131] req@ffff8800a705df80 x1818247332923136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018131.399041:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332923136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018131.399049:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018131.399052:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018131.399055:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332923136:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018131.399061:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332923264:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018131.399077:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018131.399084:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018131/real 1734018131] req@ffff8800a705c700 x1818247332923264/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018131.399094:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332923264/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018131.399101:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018131.399103:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018131.399106:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332923264:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018131.399112:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332923392:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018131.399126:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018131.399131:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018131/real 1734018131] req@ffff8800a705ed80 x1818247332923392/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018131.399142:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332923392/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018131.399150:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018131.399152:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018131.399155:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332923392:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018131.970483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018132.972522:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018133.974435:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018134.976479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018135.978460:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018136.406359:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018136.406369:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018136.406373:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018136.406384:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2398 00000100:00080000:0.0:1734018136.406394:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018136.406402:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018136.406439:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332923520/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018136.406475:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018136.406479:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018136.406481:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018136.406485:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2398 00000100:00080000:0.0:1734018136.406489:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018136.406493:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018136.406503:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018136.406514:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332923648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018136.406524:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018136.406527:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018136.406529:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018136.406532:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2398 00000100:00080000:0.0:1734018136.406535:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018136.406539:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018136.406544:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018136.406552:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332923776/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018136.406562:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018136.406565:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018136.406567:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018136.406570:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2398 00000100:00080000:0.0:1734018136.406572:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018136.406576:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018136.406580:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018136.406587:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332923904/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018136.406597:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018136.406600:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018136.406602:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018136.406605:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2398 00000100:00080000:0.0:1734018136.406608:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018136.406609:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332923520:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018136.406612:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018136.406616:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018136.406624:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332924032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018136.406632:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018136.406636:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018136.406637:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018136.406642:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2398 00000100:00080000:0.0:1734018136.406645:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018136.406648:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018136.406653:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018136.406660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332924160/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018136.406711:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018136.406714:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018136.406716:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018136.406720:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2398 00000100:00080000:0.0:1734018136.406723:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018136.406727:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018136.406731:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018136.406742:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332924288/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018136.406742:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018136.406772:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018136/real 1734018136] req@ffff8800a705e300 x1818247332923520/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018136.406802:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332923520/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018136.406815:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018136.406819:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018136.406824:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332923520:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018136.406833:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332923648:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018136.406854:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018136.406861:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018136/real 1734018136] req@ffff8800a705c000 x1818247332923648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018136.406872:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332923648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018136.406880:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018136.406883:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018136.406887:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332923648:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018136.406893:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332923776:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018136.406908:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018136.406914:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018136/real 1734018136] req@ffff8800a705d500 x1818247332923776/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018136.406924:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332923776/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018136.406931:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018136.406935:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018136.406938:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332923776:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018136.406970:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332923904:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018136.406983:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018136.406989:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018136/real 1734018136] req@ffff8800a705ea00 x1818247332923904/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018136.407000:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332923904/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018136.407008:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018136.407010:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018136.407013:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332923904:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018136.407019:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332924032:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018136.407032:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018136.407038:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018136/real 1734018136] req@ffff8800a705fb80 x1818247332924032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018136.407047:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332924032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018136.407055:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018136.407057:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018136.407060:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332924032:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018136.407067:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332924160:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018136.407084:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018136.407090:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018136/real 1734018136] req@ffff8800a705e680 x1818247332924160/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018136.407100:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332924160/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018136.407107:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018136.407110:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018136.407112:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332924160:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018136.407118:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332924288:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018136.407131:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018136.407137:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018136/real 1734018136] req@ffff8800a705ca80 x1818247332924288/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018136.407146:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332924288/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018136.407154:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018136.407156:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018136.407159:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332924288:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018136.980438:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018137.982510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018138.984507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018139.986499:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018140.988525:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018141.414376:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018141.414388:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018141.414394:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018141.414408:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2403 00000100:00080000:0.0:1734018141.414419:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018141.414429:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018141.414477:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332924416/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018141.414523:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018141.414527:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018141.414529:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018141.414534:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2403 00000100:00080000:0.0:1734018141.414538:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018141.414542:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018141.414554:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018141.414566:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332924544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018141.414576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018141.414580:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018141.414622:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018141.414627:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2403 00000100:00080000:0.0:1734018141.414630:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018141.414634:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018141.414640:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018141.414653:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332924672/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018141.414662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018141.414666:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018141.414668:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018141.414671:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2403 00000100:00100000:1.0:1734018141.414672:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332924416:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018141.414674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018141.414678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018141.414683:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018141.414692:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332924800/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018141.414700:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018141.414704:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018141.414705:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018141.414709:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2403 00000100:00080000:0.0:1734018141.414711:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018141.414715:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018141.414719:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018141.414728:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332924928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018141.414736:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018141.414739:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018141.414741:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018141.414745:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2403 00000100:00080000:0.0:1734018141.414748:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018141.414751:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018141.414756:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018141.414760:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018141.414764:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332925056/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018141.414772:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018141.414776:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018141.414778:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018141.414782:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2403 00000100:00080000:0.0:1734018141.414785:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018141.414788:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018141/real 1734018141] req@ffff8800a705f800 x1818247332924416/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018141.414789:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018141.414793:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018141.414802:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332925184/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018141.414818:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332924416/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018141.414829:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018141.414832:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018141.414837:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332924416:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018141.414845:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332924544:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018141.414865:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018141.414871:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018141/real 1734018141] req@ffff8800a705d880 x1818247332924544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018141.414883:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332924544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018141.414891:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018141.414893:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018141.414897:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332924544:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018141.414923:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332924672:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018141.414938:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018141.414944:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018141/real 1734018141] req@ffff8800a705ca80 x1818247332924672/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018141.414954:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332924672/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018141.414962:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018141.414966:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018141.414969:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332924672:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018141.414975:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332924800:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018141.414988:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018141.414993:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018141/real 1734018141] req@ffff8800a705e680 x1818247332924800/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018141.415003:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332924800/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018141.415010:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018141.415012:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018141.415015:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332924800:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018141.415021:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332924928:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018141.415034:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018141.415039:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018141/real 1734018141] req@ffff8800a705fb80 x1818247332924928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018141.415050:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332924928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018141.415058:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018141.415060:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018141.415063:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332924928:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018141.415069:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332925056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018141.415084:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018141.415090:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018141/real 1734018141] req@ffff8800a705ea00 x1818247332925056/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018141.415101:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332925056/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018141.415108:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018141.415110:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018141.415113:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332925056:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018141.415118:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332925184:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018141.415130:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018141.415136:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018141/real 1734018141] req@ffff8800a705d500 x1818247332925184/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018141.415145:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332925184/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018141.415152:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018141.415155:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018141.415158:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332925184:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018141.990471:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018142.992550:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018143.994517:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018144.996447:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018145.998481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018146.422342:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018146.422353:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018146.422358:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018146.422373:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2408 00000100:00080000:0.0:1734018146.422384:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018146.422395:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018146.422442:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332925312/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018146.422479:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018146.422484:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018146.422486:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018146.422492:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2408 00000100:00080000:0.0:1734018146.422535:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018146.422541:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018146.422554:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018146.422567:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332925440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018146.422579:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018146.422583:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018146.422585:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018146.422589:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2408 00000100:00100000:1.0:1734018146.422590:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332925312:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018146.422593:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018146.422596:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018146.422601:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018146.422610:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332925568/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018146.422619:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018146.422622:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018146.422624:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018146.422627:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2408 00000100:00080000:0.0:1734018146.422630:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018146.422634:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018146.422638:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018146.422647:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332925696/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018146.422655:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018146.422658:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018146.422660:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018146.422663:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2408 00000100:00080000:0.0:1734018146.422667:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018146.422670:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018146.422675:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1734018146.422677:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018146.422684:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332925824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018146.422692:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018146.422696:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018146.422697:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018146.422701:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2408 00000100:00080000:0.0:1734018146.422704:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018146.422706:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018146/real 1734018146] req@ffff8800a705c000 x1818247332925312/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018146.422708:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018146.422712:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018146.422720:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332925952/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018146.422728:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018146.422732:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018146.422733:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018146.422736:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332925312/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018146.422737:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2408 00000100:00080000:0.0:1734018146.422739:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018146.422743:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018146.422747:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734018146.422747:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018146.422750:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018146.422756:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332925312:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018146.422758:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332926080/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018146.422782:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332925440:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018146.422801:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018146.422807:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018146/real 1734018146] req@ffff8800a705e300 x1818247332925440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018146.422819:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332925440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018146.422826:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018146.422829:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018146.422832:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332925440:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018146.422838:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332925568:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018146.422853:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018146.422858:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018146/real 1734018146] req@ffff8800a705ed80 x1818247332925568/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018146.422868:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332925568/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018146.422876:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018146.422879:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018146.422882:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332925568:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018146.422889:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332925696:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018146.422902:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018146.422907:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018146/real 1734018146] req@ffff8800a705c700 x1818247332925696/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018146.422917:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332925696/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018146.422925:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018146.422928:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018146.422931:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332925696:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018146.422936:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332925824:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018146.422949:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018146.422955:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018146/real 1734018146] req@ffff8800a705df80 x1818247332925824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018146.422964:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332925824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018146.422972:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018146.422975:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018146.422978:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332925824:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018146.422984:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332925952:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018146.423001:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018146.423007:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018146/real 1734018146] req@ffff8800a705dc00 x1818247332925952/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018146.423018:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332925952/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018146.423025:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018146.423027:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018146.423030:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332925952:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018146.423035:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332926080:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018146.423047:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018146.423053:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018146/real 1734018146] req@ffff8800a705f100 x1818247332926080/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018146.423063:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332926080/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018146.423070:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018146.423072:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018146.423075:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332926080:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018147.000602:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018148.002575:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018149.004538:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018150.006504:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018151.008499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018151.430479:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018151.430489:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018151.430494:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018151.430505:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2413 00000100:00080000:2.0:1734018151.430514:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018151.430521:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018151.430558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247332926208/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018151.430597:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018151.430601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018151.430603:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018151.430607:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2413 00000100:00080000:2.0:1734018151.430610:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018151.430615:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018151.430625:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018151.430636:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247332926336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018151.430646:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018151.430650:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018151.430651:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018151.430655:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2413 00000100:00080000:2.0:1734018151.430658:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018151.430661:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018151.430666:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018151.430676:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247332926464/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018151.430684:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018151.430687:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018151.430689:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018151.430692:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2413 00000100:00080000:2.0:1734018151.430694:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018151.430698:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018151.430702:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018151.430711:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247332926592/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018151.430719:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018151.430723:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018151.430725:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018151.430728:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2413 00000100:00080000:2.0:1734018151.430731:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018151.430734:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018151.430738:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018151.430746:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247332926720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018151.430756:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018151.430759:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018151.430760:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018151.430765:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332926208:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734018151.430765:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2413 00000100:00080000:2.0:1734018151.430767:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018151.430771:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018151.430775:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018151.430783:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247332926848/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018151.430791:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018151.430794:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018151.430796:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018151.430799:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2413 00000100:00080000:2.0:1734018151.430802:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018151.430805:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018151.430810:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018151.430818:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247332926976/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018151.430862:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018151.430891:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018151/real 1734018151] req@ffff8800a73ca680 x1818247332926208/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018151.430922:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ca680 x1818247332926208/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018151.430934:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018151.430938:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018151.430943:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332926208:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018151.430951:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332926336:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018151.430970:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018151.430976:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018151/real 1734018151] req@ffff8800a73c9f80 x1818247332926336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018151.430987:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9f80 x1818247332926336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018151.430995:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018151.430998:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018151.431001:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332926336:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018151.431008:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332926464:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018151.431024:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018151.431030:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018151/real 1734018151] req@ffff8800a73caa00 x1818247332926464/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018151.431040:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73caa00 x1818247332926464/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018151.431047:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018151.431051:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018151.431054:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332926464:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018151.431061:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332926592:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018151.431074:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018151.431080:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018151/real 1734018151] req@ffff8800a73c8e00 x1818247332926592/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018151.431090:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247332926592/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018151.431098:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018151.431100:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018151.431103:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332926592:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018151.431136:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332926720:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018151.431149:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018151.431155:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018151/real 1734018151] req@ffff8800a73c8000 x1818247332926720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018151.431165:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247332926720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018151.431174:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018151.431176:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018151.431179:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332926720:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018151.431185:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332926848:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018151.431201:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018151.431207:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018151/real 1734018151] req@ffff8800a73c9500 x1818247332926848/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018151.431280:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247332926848/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018151.431287:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018151.431290:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018151.431293:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332926848:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018151.431301:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332926976:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018151.431317:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018151.431324:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018151/real 1734018151] req@ffff8800a73c9c00 x1818247332926976/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018151.431334:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247332926976/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018151.431343:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018151.431345:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018151.431348:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332926976:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018152.010477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018153.012480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018154.014541:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018155.016547:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018156.018528:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018156.438390:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018156.438401:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018156.438406:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018156.438416:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2418 00000100:00080000:2.0:1734018156.438425:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018156.438433:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018156.438469:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247332927104/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018156.438503:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018156.438507:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018156.438509:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018156.438514:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2418 00000100:00080000:2.0:1734018156.438517:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018156.438521:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018156.438531:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018156.438541:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247332927232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018156.438551:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018156.438557:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018156.438559:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018156.438563:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2418 00000100:00080000:2.0:1734018156.438565:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018156.438569:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018156.438573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018156.438581:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247332927360/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018156.438590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018156.438593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018156.438594:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018156.438598:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2418 00000100:00080000:2.0:1734018156.438601:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018156.438604:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018156.438608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018156.438617:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247332927488/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018156.438625:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018156.438629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018156.438630:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018156.438634:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2418 00000100:00080000:2.0:1734018156.438636:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018156.438639:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018156.438644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018156.438653:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247332927616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018156.438662:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332927104:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734018156.438663:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018156.438666:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018156.438668:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018156.438671:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2418 00000100:00080000:2.0:1734018156.438674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018156.438677:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018156.438682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018156.438689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247332927744/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018156.438697:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018156.438700:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018156.438702:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018156.438705:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2418 00000100:00080000:2.0:1734018156.438708:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018156.438711:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018156.438715:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018156.438723:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247332927872/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018156.438759:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018156.438789:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018156/real 1734018156] req@ffff8800a73cad80 x1818247332927104/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018156.438820:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247332927104/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018156.438833:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018156.438836:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018156.438841:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332927104:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018156.438849:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332927232:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018156.438868:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018156.438875:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018156/real 1734018156] req@ffff8800a73cb100 x1818247332927232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018156.438886:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247332927232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018156.438894:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018156.438897:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018156.438900:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332927232:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018156.438907:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332927360:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018156.438923:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018156.438929:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018156/real 1734018156] req@ffff8800a73c9880 x1818247332927360/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018156.438939:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247332927360/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018156.438946:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018156.438950:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018156.438953:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332927360:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018156.438960:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332927488:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018156.438973:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018156.438979:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018156/real 1734018156] req@ffff8800a73c9c00 x1818247332927488/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018156.438988:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247332927488/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018156.438996:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018156.438998:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018156.439001:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332927488:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018156.439033:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332927616:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018156.439046:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018156.439052:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018156/real 1734018156] req@ffff8800a73c9500 x1818247332927616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018156.439062:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247332927616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018156.439070:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018156.439072:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018156.439075:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332927616:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018156.439081:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332927744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018156.439099:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018156.439105:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018156/real 1734018156] req@ffff8800a73c8000 x1818247332927744/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018156.439115:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247332927744/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018156.439122:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018156.439124:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018156.439127:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332927744:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018156.439132:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332927872:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018156.439145:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018156.439151:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018156/real 1734018156] req@ffff8800a73c8e00 x1818247332927872/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018156.439161:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247332927872/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018156.439168:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018156.439171:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018156.439174:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332927872:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018157.020566:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018158.022494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018159.024573:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018160.026497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018161.028485:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018161.446363:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018161.446374:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018161.446378:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018161.446389:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2423 00000100:00080000:2.0:1734018161.446404:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018161.446412:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018161.446448:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247332928000/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018161.446483:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018161.446487:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018161.446489:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018161.446493:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2423 00000100:00080000:2.0:1734018161.446497:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018161.446501:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018161.446511:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018161.446521:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247332928128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018161.446531:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018161.446535:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018161.446536:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018161.446540:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2423 00000100:00080000:2.0:1734018161.446543:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018161.446546:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018161.446551:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018161.446561:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247332928256/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018161.446571:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018161.446574:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018161.446576:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018161.446579:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2423 00000100:00080000:2.0:1734018161.446582:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018161.446585:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018161.446590:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018161.446598:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247332928384/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018161.446607:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018161.446611:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018161.446612:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018161.446616:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2423 00000100:00080000:2.0:1734018161.446619:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018161.446622:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332928000:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734018161.446622:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018161.446627:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018161.446635:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247332928512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018161.446644:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018161.446647:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018161.446649:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018161.446653:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2423 00000100:00080000:2.0:1734018161.446657:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018161.446660:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018161.446664:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018161.446672:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247332928640/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018161.446680:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018161.446683:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018161.446684:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018161.446688:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2423 00000100:00080000:2.0:1734018161.446690:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018161.446694:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018161.446698:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018161.446705:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247332928768/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018161.446720:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018161.446750:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018161/real 1734018161] req@ffff8800a73caa00 x1818247332928000/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018161.446781:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73caa00 x1818247332928000/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018161.446794:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018161.446798:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018161.446803:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332928000:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018161.446811:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332928128:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018161.446830:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018161.446837:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018161/real 1734018161] req@ffff8800a73c9f80 x1818247332928128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018161.446848:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9f80 x1818247332928128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018161.446856:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018161.446859:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018161.446862:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332928128:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018161.446869:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332928256:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018161.446885:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018161.446890:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018161/real 1734018161] req@ffff8800a73ca680 x1818247332928256/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018161.446900:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ca680 x1818247332928256/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018161.446908:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018161.446912:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018161.446915:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332928256:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018161.446946:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332928384:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018161.446960:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018161.446966:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018161/real 1734018161] req@ffff8800a73c8e00 x1818247332928384/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018161.446978:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247332928384/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018161.446985:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018161.446988:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018161.446991:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332928384:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018161.446997:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332928512:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018161.447010:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018161.447015:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018161/real 1734018161] req@ffff8800a73c8000 x1818247332928512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018161.447025:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247332928512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018161.447033:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018161.447035:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018161.447039:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332928512:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018161.447045:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332928640:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018161.447062:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018161.447068:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018161/real 1734018161] req@ffff8800a73c9500 x1818247332928640/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018161.447079:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247332928640/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018161.447086:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018161.447088:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018161.447091:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332928640:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018161.447097:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332928768:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018161.447110:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018161.447115:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018161/real 1734018161] req@ffff8800a73c9c00 x1818247332928768/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018161.447126:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247332928768/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018161.447134:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018161.447136:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018161.447139:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332928768:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018162.030480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018163.032477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018164.034531:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018165.036482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018166.038463:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018166.454353:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018166.454362:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018166.454367:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018166.454377:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2428 00000100:00080000:2.0:1734018166.454387:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018166.454394:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018166.454430:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247332928896/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018166.454473:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018166.454477:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018166.454479:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018166.454484:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2428 00000100:00080000:2.0:1734018166.454487:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018166.454491:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018166.454500:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018166.454511:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247332929024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018166.454521:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018166.454525:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018166.454526:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018166.454530:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2428 00000100:00080000:2.0:1734018166.454533:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018166.454536:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018166.454541:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018166.454549:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247332929152/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018166.454558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018166.454561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018166.454562:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018166.454565:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2428 00000100:00080000:2.0:1734018166.454568:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018166.454572:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018166.454576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018166.454585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247332929280/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018166.454594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018166.454598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018166.454599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018166.454603:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2428 00000100:00080000:2.0:1734018166.454605:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018166.454609:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018166.454611:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332928896:192.168.201.152@tcp:250:kworker.0 00000008:00100000:2.0:1734018166.454613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018166.454622:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247332929408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018166.454630:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018166.454633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018166.454634:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018166.454638:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2428 00000100:00080000:2.0:1734018166.454641:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018166.454645:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018166.454649:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018166.454657:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247332929536/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018166.454664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018166.454668:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018166.454670:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018166.454673:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2428 00000100:00080000:2.0:1734018166.454676:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018166.454679:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018166.454683:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018166.454690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247332929664/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018166.454709:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018166.454738:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018166/real 1734018166] req@ffff8800a73c9880 x1818247332928896/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018166.454769:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247332928896/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018166.454782:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018166.454785:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018166.454791:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332928896:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018166.454799:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332929024:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018166.454818:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018166.454824:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018166/real 1734018166] req@ffff8800a73cb100 x1818247332929024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018166.454836:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247332929024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018166.454844:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018166.454847:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018166.454850:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332929024:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018166.454857:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332929152:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018166.454872:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018166.454878:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018166/real 1734018166] req@ffff8800a73cad80 x1818247332929152/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018166.454888:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247332929152/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018166.454896:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018166.454899:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018166.454902:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332929152:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018166.454934:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332929280:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018166.454948:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018166.454954:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018166/real 1734018166] req@ffff8800a73c9c00 x1818247332929280/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018166.454965:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247332929280/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018166.454973:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018166.454975:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018166.454978:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332929280:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018166.454984:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332929408:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018166.454997:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018166.455002:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018166/real 1734018166] req@ffff8800a73c9500 x1818247332929408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018166.455012:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247332929408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018166.455020:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018166.455022:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018166.455025:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332929408:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018166.455032:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332929536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018166.455048:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018166.455054:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018166/real 1734018166] req@ffff8800a73c8000 x1818247332929536/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018166.455065:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247332929536/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018166.455072:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018166.455074:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018166.455077:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332929536:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018166.455083:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332929664:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018166.455095:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018166.455101:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018166/real 1734018166] req@ffff8800a73c8e00 x1818247332929664/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018166.455111:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247332929664/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018166.455118:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018166.455121:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018166.455123:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332929664:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018167.040503:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018168.042493:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018169.044439:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018170.046458:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018171.048478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018171.462448:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018171.462457:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018171.462462:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018171.462472:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2434 00000100:00080000:0.0:1734018171.462481:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018171.462489:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018171.462524:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332929792/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018171.462562:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018171.462566:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018171.462568:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018171.462573:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2434 00000100:00080000:0.0:1734018171.462576:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018171.462580:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018171.462590:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018171.462601:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332929920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018171.462611:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018171.462614:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018171.462616:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018171.462620:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2434 00000100:00080000:0.0:1734018171.462622:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018171.462625:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018171.462631:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018171.462641:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332930048/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018171.462650:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018171.462653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018171.462654:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018171.462657:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2434 00000100:00080000:0.0:1734018171.462660:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018171.462663:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018171.462668:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018171.462671:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332929792:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018171.462676:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332930176/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018171.462685:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018171.462688:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018171.462689:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018171.462693:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2434 00000100:00080000:0.0:1734018171.462695:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018171.462699:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018171.462703:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018171.462711:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332930304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018171.462719:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018171.462723:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018171.462724:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018171.462728:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2434 00000100:00080000:0.0:1734018171.462731:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018171.462734:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018171.462739:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018171.462739:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018171.462746:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332930432/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018171.462754:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018171.462757:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018171.462758:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018171.462761:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018171/real 1734018171] req@ffff8800a705d180 x1818247332929792/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018171.462762:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2434 00000100:00080000:0.0:1734018171.462764:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018171.462767:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018171.462772:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018171.462779:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332930560/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018171.462788:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332929792/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018171.462798:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018171.462801:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018171.462805:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332929792:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018171.462813:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332929920:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018171.462829:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018171.462836:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018171/real 1734018171] req@ffff8800a705c380 x1818247332929920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018171.462847:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332929920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018171.462855:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018171.462857:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018171.462861:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332929920:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018171.462867:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332930048:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018171.462880:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018171.462885:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018171/real 1734018171] req@ffff8800a705f100 x1818247332930048/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018171.462895:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332930048/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018171.462903:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018171.462906:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018171.462909:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332930048:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018171.462937:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332930176:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018171.462950:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018171.462956:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018171/real 1734018171] req@ffff8800a705dc00 x1818247332930176/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018171.462966:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332930176/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018171.462974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018171.462976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018171.462979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332930176:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018171.462984:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332930304:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018171.462996:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018171.463001:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018171/real 1734018171] req@ffff8800a705df80 x1818247332930304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018171.463011:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332930304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018171.463018:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018171.463021:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018171.463024:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332930304:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018171.463029:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332930432:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018171.463044:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018171.463050:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018171/real 1734018171] req@ffff8800a705c700 x1818247332930432/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018171.463060:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332930432/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018171.463067:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018171.463117:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018171.463121:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332930432:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018171.463128:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332930560:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018171.463143:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018171.463151:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018171/real 1734018171] req@ffff8800a705ed80 x1818247332930560/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018171.463161:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332930560/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018171.463169:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018171.463171:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018171.463174:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332930560:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018172.050484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018173.052537:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018174.054489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018175.056514:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018176.058488:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018176.470336:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018176.470345:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018176.470349:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018176.470359:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2439 00000100:00080000:0.0:1734018176.470369:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018176.470376:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018176.470411:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332930688/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018176.470452:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018176.470457:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018176.470459:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018176.470463:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2439 00000100:00080000:0.0:1734018176.470467:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018176.470470:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018176.470480:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018176.470491:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332930816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018176.470502:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018176.470505:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018176.470507:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018176.470511:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2439 00000100:00080000:0.0:1734018176.470513:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018176.470515:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332930688:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018176.470517:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018176.470522:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018176.470530:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332930944/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018176.470539:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018176.470542:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018176.470543:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018176.470546:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2439 00000100:00080000:0.0:1734018176.470549:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018176.470553:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018176.470557:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018176.470565:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332931072/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018176.470573:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018176.470576:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018176.470578:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018176.470581:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2439 00000800:00000100:1.0:1734018176.470582:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018176.470584:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018176.470587:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018176.470591:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018176.470599:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332931200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018176.470604:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018176/real 1734018176] req@ffff8800a705e300 x1818247332930688/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018176.470607:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018176.470610:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018176.470611:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018176.470615:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2439 00000100:00080000:0.0:1734018176.470618:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018176.470621:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018176.470625:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018176.470631:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332930688/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018176.470633:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332931328/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018176.470641:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1734018176.470641:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018176.470644:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:0.0:1734018176.470645:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018176.470646:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018176.470649:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332930688:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018176.470650:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2439 00000100:00080000:0.0:1734018176.470652:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018176.470656:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018176.470660:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018176.470668:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332931456/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018176.470670:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332930816:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018176.470687:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018176.470694:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018176/real 1734018176] req@ffff8800a705c000 x1818247332930816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018176.470705:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332930816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018176.470713:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018176.470715:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018176.470719:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332930816:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018176.470725:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332930944:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018176.470738:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018176.470744:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018176/real 1734018176] req@ffff8800a705d500 x1818247332930944/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018176.470754:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332930944/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018176.470762:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018176.470764:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018176.470768:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332930944:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018176.470773:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332931072:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018176.470785:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018176.470791:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018176/real 1734018176] req@ffff8800a705ea00 x1818247332931072/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018176.470800:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332931072/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018176.470808:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018176.470810:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018176.470813:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332931072:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018176.470818:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332931200:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018176.470831:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018176.470837:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018176/real 1734018176] req@ffff8800a705fb80 x1818247332931200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018176.470848:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332931200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018176.470855:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018176.470857:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018176.470860:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332931200:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018176.470865:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332931328:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018176.470880:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018176.470886:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018176/real 1734018176] req@ffff8800a705e680 x1818247332931328/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018176.470897:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332931328/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018176.470904:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018176.470906:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018176.470909:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332931328:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018176.470926:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332931456:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018176.470937:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018176.470943:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018176/real 1734018176] req@ffff8800a705ca80 x1818247332931456/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018176.470953:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332931456/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018176.470960:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018176.470963:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018176.470965:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332931456:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018177.060457:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018178.062477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018179.064468:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018180.066508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018181.068415:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018181.478355:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018181.478368:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018181.478373:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018181.478388:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2444 00000100:00080000:0.0:1734018181.478399:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018181.478410:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018181.478459:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332931584/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018181.478497:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018181.478502:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018181.478504:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018181.478510:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2444 00000100:00080000:0.0:1734018181.478513:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018181.478517:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018181.478529:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018181.478541:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332931712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018181.478552:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018181.478556:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018181.478557:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018181.478561:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2444 00000100:00080000:0.0:1734018181.478564:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018181.478567:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018181.478572:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018181.478583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332931840/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018181.478592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018181.478595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018181.478597:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018181.478600:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2444 00000100:00080000:0.0:1734018181.478602:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018181.478606:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018181.478611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018181.478621:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332931968/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018181.478621:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332931584:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018181.478631:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018181.478634:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018181.478636:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018181.478639:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2444 00000100:00080000:0.0:1734018181.478643:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018181.478646:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018181.478651:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018181.478660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332932096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018181.478668:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018181.478671:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018181.478673:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018181.478677:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2444 00000100:00080000:0.0:1734018181.478681:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018181.478685:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018181.478689:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018181.478695:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018181.478697:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332932224/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018181.478706:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018181.478709:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018181.478710:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018181.478714:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2444 00000100:00080000:0.0:1734018181.478717:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018181.478720:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018181.478720:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018181/real 1734018181] req@ffff8800a705d880 x1818247332931584/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018181.478725:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018181.478733:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332932352/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018181.478748:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332931584/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018181.478759:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018181.478763:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018181.478768:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332931584:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018181.478777:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332931712:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018181.478796:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018181.478803:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018181/real 1734018181] req@ffff8800a705f800 x1818247332931712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018181.478814:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332931712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018181.478822:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018181.478825:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018181.478828:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332931712:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018181.478834:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332931840:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018181.478848:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018181.478855:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018181/real 1734018181] req@ffff8800a705ca80 x1818247332931840/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018181.478865:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332931840/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018181.478873:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018181.478876:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018181.478879:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332931840:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018181.478938:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332931968:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018181.478955:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018181.478963:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018181/real 1734018181] req@ffff8800a705e680 x1818247332931968/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018181.478973:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332931968/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018181.478981:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018181.478983:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018181.478986:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332931968:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018181.478992:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332932096:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018181.479005:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018181.479011:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018181/real 1734018181] req@ffff8800a705fb80 x1818247332932096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018181.479020:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332932096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018181.479029:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018181.479031:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018181.479035:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332932096:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018181.479040:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332932224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018181.479057:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018181.479063:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018181/real 1734018181] req@ffff8800a705ea00 x1818247332932224/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018181.479074:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332932224/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018181.479082:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018181.479084:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018181.479087:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332932224:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018181.479093:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332932352:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018181.479105:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018181.479110:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018181/real 1734018181] req@ffff8800a705d500 x1818247332932352/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018181.479120:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332932352/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018181.479127:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018181.479129:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018181.479132:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332932352:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018182.070470:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018183.072514:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018184.074499:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018185.076422:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018186.078452:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018186.486346:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018186.486355:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018186.486360:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018186.486371:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2449 00000100:00080000:0.0:1734018186.486380:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018186.486388:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018186.486424:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332932480/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018186.486458:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018186.486462:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018186.486464:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018186.486469:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2449 00000100:00080000:0.0:1734018186.486472:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018186.486476:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018186.486486:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018186.486497:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332932608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018186.486507:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018186.486510:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018186.486512:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018186.486515:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2449 00000100:00080000:0.0:1734018186.486518:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018186.486521:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018186.486527:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018186.486534:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332932736/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018186.486543:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018186.486546:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018186.486548:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018186.486551:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2449 00000100:00080000:0.0:1734018186.486554:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018186.486557:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018186.486561:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018186.486569:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332932864/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018186.486573:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332932480:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018186.486578:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018186.486581:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018186.486583:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018186.486586:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2449 00000100:00080000:0.0:1734018186.486589:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018186.486592:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018186.486596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018186.486604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332932992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018186.486612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018186.486615:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018186.486618:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018186.486621:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2449 00000100:00080000:0.0:1734018186.486626:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018186.486629:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018186.486633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018186.486640:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332933120/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018186.486648:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018186.486651:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018186.486653:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018186.486656:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2449 00000100:00080000:0.0:1734018186.486658:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018186.486662:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018186.486665:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332932608:192.168.201.152@tcp:38:kworker.0 00000008:00100000:0.0:1734018186.486666:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018186.486674:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332933248/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018186.486686:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332932736:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018186.486709:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332932864:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018186.486726:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332932992:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018186.486741:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332933120:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018186.486755:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332933248:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018187.080464:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018188.082479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018189.084349:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1734018189.492576:0:1956:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:2.0:1734018189.492657:0:1956:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1734018189.492686:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018189.492747:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018189.492762:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018189.492818:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018189.492830:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018189.492842:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018189.492856:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734018189.492861:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018186/real 1734018189] req@ffff8800a705c700 x1818247332932864/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018189.492895:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332932864/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018189.492908:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018189.492912:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018189.492917:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332932864:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018189.492925:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018186/real 1734018189] req@ffff8800a705df80 x1818247332932992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018189.492938:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332932992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018189.492946:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018189.492949:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018189.492952:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332932992:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018189.492959:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018186/real 1734018189] req@ffff8800a705dc00 x1818247332933120/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018189.492970:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332933120/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018189.492980:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018189.492983:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018189.492986:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332933120:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018189.492992:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018186/real 1734018189] req@ffff8800a705f100 x1818247332933248/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018189.493003:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332933248/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018189.493010:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018189.493013:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018189.493016:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332933248:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018189.493022:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018186/real 1734018189] req@ffff8800a705c000 x1818247332932480/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018189.493034:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332932480/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018189.493042:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018189.493044:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018189.493047:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332932480:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018189.493053:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018186/real 1734018189] req@ffff8800a705e300 x1818247332932608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018189.493063:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332932608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018189.493078:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018189.493081:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018189.493084:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332932608:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018189.493090:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018186/real 1734018189] req@ffff8800a705ed80 x1818247332932736/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018189.493100:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332932736/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018189.493108:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018189.493110:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018189.493113:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332932736:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018190.086419:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018191.088373:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018191.494341:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018191.494350:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018191.494354:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018191.494365:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2454 00000100:00080000:0.0:1734018191.494381:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018191.494389:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018191.494425:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332933376/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018191.494468:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018191.494472:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018191.494474:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018191.494479:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2454 00000100:00080000:0.0:1734018191.494482:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018191.494486:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018191.494496:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018191.494507:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332933504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018191.494517:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018191.494520:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018191.494522:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018191.494526:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2454 00000100:00080000:0.0:1734018191.494528:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018191.494532:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018191.494536:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018191.494546:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332933632/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018191.494554:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018191.494558:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018191.494559:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018191.494562:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2454 00000100:00080000:0.0:1734018191.494565:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018191.494569:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018191.494573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018191.494581:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332933760/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018191.494583:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332933376:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018191.494590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018191.494593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018191.494595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018191.494598:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2454 00000100:00080000:0.0:1734018191.494601:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018191.494604:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018191.494608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018191.494615:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332933888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018191.494623:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018191.494626:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018191.494628:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018191.494631:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2454 00000100:00080000:0.0:1734018191.494636:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018191.494640:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018191.494644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018191.494651:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332934016/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018191.494651:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018191.494660:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018191.494664:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018191.494665:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018191.494669:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2454 00000100:00080000:0.0:1734018191.494672:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018191.494673:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018191/real 1734018191] req@ffff8800a705c380 x1818247332933376/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018191.494675:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018191.494679:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018191.494687:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332934144/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018191.494692:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332933376/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018191.494703:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018191.494706:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018191.494711:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332933376:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018191.494718:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332933504:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018191.494782:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018191.494792:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018191/real 1734018191] req@ffff8800a705d180 x1818247332933504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018191.494803:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332933504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018191.494811:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018191.494814:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018191.494817:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332933504:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018191.494823:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332933632:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018191.494838:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018191.494844:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018191/real 1734018191] req@ffff8800a705ed80 x1818247332933632/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018191.494854:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332933632/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018191.494862:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018191.494864:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018191.494867:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332933632:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018191.494892:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332933760:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018191.494904:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018191.494911:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018191/real 1734018191] req@ffff8800a705e300 x1818247332933760/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018191.494920:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332933760/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018191.494935:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018191.494938:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018191.494941:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332933760:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018191.494946:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332933888:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018191.494957:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018191.494963:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018191/real 1734018191] req@ffff8800a705c000 x1818247332933888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018191.494973:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332933888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018191.494981:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018191.494983:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018191.494986:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332933888:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018191.494992:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332934016:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018191.495003:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018191.495009:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018191/real 1734018191] req@ffff8800a705f100 x1818247332934016/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018191.495019:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332934016/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018191.495026:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018191.495028:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018191.495032:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332934016:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018191.495038:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332934144:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018191.495050:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018191.495056:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018191/real 1734018191] req@ffff8800a705dc00 x1818247332934144/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018191.495065:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332934144/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018191.495073:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018191.495075:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018191.495078:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332934144:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018192.090372:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018193.092360:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018194.094350:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018195.096476:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018196.098515:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018196.502346:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018196.502358:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018196.502363:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018196.502378:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2459 00000100:00080000:0.0:1734018196.502389:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018196.502399:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018196.502447:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332934272/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018196.502486:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018196.502491:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018196.502493:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018196.502499:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2459 00000100:00080000:0.0:1734018196.502502:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018196.502506:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018196.502518:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018196.502529:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332934400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018196.502541:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018196.502545:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018196.502546:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018196.502550:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2459 00000100:00080000:0.0:1734018196.502553:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018196.502556:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018196.502561:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018196.502570:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332934528/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018196.502579:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018196.502582:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018196.502584:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018196.502587:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2459 00000100:00080000:0.0:1734018196.502590:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018196.502593:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018196.502598:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018196.502607:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332934656/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018196.502617:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018196.502620:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018196.502622:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018196.502626:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2459 00000100:00100000:1.0:1734018196.502627:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332934272:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018196.502629:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018196.502633:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018196.502638:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018196.502682:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332934784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018196.502692:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018196.502695:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018196.502697:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018196.502701:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2459 00000100:00080000:0.0:1734018196.502703:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018196.502707:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018196.502711:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018196.502720:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332934912/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018196.502728:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018196.502732:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018196.502734:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018196.502737:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2459 00000100:00080000:0.0:1734018196.502740:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018196.502743:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018196.502747:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734018196.502748:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018196.502759:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332935040/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018196.502771:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018196/real 1734018196] req@ffff8800a705df80 x1818247332934272/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018196.502791:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332934272/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018196.502804:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018196.502807:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018196.502812:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332934272:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018196.502820:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332934400:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018196.502839:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018196.502846:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018196/real 1734018196] req@ffff8800a705c700 x1818247332934400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018196.502865:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332934400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018196.502873:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018196.502876:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018196.502880:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332934400:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018196.502886:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332934528:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018196.502899:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018196.502905:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018196/real 1734018196] req@ffff8800a705d500 x1818247332934528/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018196.502916:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332934528/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018196.502923:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018196.502925:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018196.502928:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332934528:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018196.502954:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332934656:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018196.502967:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018196.502973:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018196/real 1734018196] req@ffff8800a705ea00 x1818247332934656/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018196.502984:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332934656/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018196.502992:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018196.502995:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018196.502998:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332934656:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018196.503004:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332934784:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018196.503016:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018196.503022:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018196/real 1734018196] req@ffff8800a705fb80 x1818247332934784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018196.503032:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332934784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018196.503040:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018196.503043:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018196.503046:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332934784:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018196.503052:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332934912:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018196.503064:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018196.503070:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018196/real 1734018196] req@ffff8800a705e680 x1818247332934912/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018196.503079:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332934912/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018196.503086:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018196.503089:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018196.503092:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332934912:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018196.503098:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332935040:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018196.503114:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018196.503120:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018196/real 1734018196] req@ffff8800a705ca80 x1818247332935040/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018196.503130:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332935040/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018196.503137:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018196.503139:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018196.503142:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332935040:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018197.100515:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018198.102476:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018199.104500:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018200.106501:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018201.108483:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018201.510369:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018201.510381:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018201.510386:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018201.510401:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2464 00000100:00080000:0.0:1734018201.510412:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018201.510422:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018201.510471:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332935168/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018201.510510:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018201.510515:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018201.510517:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018201.510522:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2464 00000100:00080000:0.0:1734018201.510525:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018201.510530:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018201.510542:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018201.510554:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332935296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018201.510608:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018201.510612:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018201.510614:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018201.510619:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2464 00000100:00080000:0.0:1734018201.510622:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018201.510626:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018201.510633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018201.510647:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332935424/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018201.510653:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332935168:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018201.510658:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018201.510662:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018201.510663:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018201.510667:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2464 00000100:00080000:0.0:1734018201.510670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018201.510673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018201.510678:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018201.510687:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332935552/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018201.510695:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018201.510698:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018201.510700:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018201.510704:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2464 00000100:00080000:0.0:1734018201.510716:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018201.510720:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018201.510725:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1734018201.510726:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018201.510734:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332935680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018201.510744:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018201.510747:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018201.510749:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018201.510750:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018201/real 1734018201] req@ffff8800a705f800 x1818247332935168/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018201.510754:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2464 00000100:00080000:0.0:1734018201.510757:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018201.510761:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018201.510766:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018201.510770:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332935168/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018201.510774:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332935808/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018201.510782:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018201.510783:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1734018201.510785:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:0.0:1734018201.510786:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018201.510788:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018201.510790:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332935168:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018201.510791:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2464 00000100:00080000:0.0:1734018201.510794:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018201.510797:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018201.510799:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332935296:192.168.201.152@tcp:38:kworker.0 00000008:00100000:0.0:1734018201.510802:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018201.510812:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332935936/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018201.510818:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018201.510826:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018201/real 1734018201] req@ffff8800a705d880 x1818247332935296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018201.510837:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332935296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018201.510846:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018201.510849:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018201.510853:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332935296:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018201.510878:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332935424:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018201.510893:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018201.510900:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018201/real 1734018201] req@ffff8800a705ca80 x1818247332935424/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018201.510910:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332935424/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018201.510918:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018201.510920:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018201.510924:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332935424:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018201.510930:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332935552:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018201.510942:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018201.510948:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018201/real 1734018201] req@ffff8800a705e680 x1818247332935552/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018201.510958:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332935552/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018201.510965:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018201.510968:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018201.510971:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332935552:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018201.510978:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332935680:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018201.510991:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018201.510997:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018201/real 1734018201] req@ffff8800a705fb80 x1818247332935680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018201.511007:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332935680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018201.511015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018201.511017:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018201.511020:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332935680:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018201.511026:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332935808:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018201.511039:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018201.511045:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018201/real 1734018201] req@ffff8800a705ea00 x1818247332935808/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018201.511055:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332935808/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018201.511062:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018201.511064:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018201.511067:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332935808:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018201.511073:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332935936:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018201.511086:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018201.511092:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018201/real 1734018201] req@ffff8800a705d500 x1818247332935936/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018201.511102:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332935936/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018201.511110:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018201.511112:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018201.511115:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332935936:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018202.110472:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018203.112595:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018204.114478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018205.116546:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018206.118529:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018206.518338:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018206.518351:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018206.518357:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018206.518370:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2469 00000100:00080000:0.0:1734018206.518382:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018206.518392:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018206.518438:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332936064/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018206.518519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018206.518525:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018206.518527:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018206.518532:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2469 00000100:00080000:0.0:1734018206.518536:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018206.518540:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018206.518552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018206.518566:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332936192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018206.518577:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018206.518581:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018206.518582:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018206.518586:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2469 00000100:00080000:0.0:1734018206.518589:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018206.518592:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018206.518597:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018206.518606:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332936320/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018206.518616:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018206.518619:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018206.518621:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018206.518624:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2469 00000100:00080000:0.0:1734018206.518627:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018206.518629:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332936064:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018206.518632:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018206.518637:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018206.518646:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332936448/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018206.518654:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018206.518657:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018206.518659:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018206.518663:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2469 00000100:00080000:0.0:1734018206.518665:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018206.518669:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018206.518673:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018206.518682:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332936576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018206.518690:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018206.518693:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018206.518694:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018206.518698:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2469 00000100:00080000:0.0:1734018206.518701:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018206.518705:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018206.518705:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734018206.518710:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018206.518717:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332936704/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018206.518727:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018206.518730:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018206/real 1734018206] req@ffff8800a705c700 x1818247332936064/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018206.518731:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018206.518733:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018206.518737:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2469 00000100:00080000:0.0:1734018206.518741:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018206.518744:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018206.518748:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018206.518750:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332936064/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018206.518760:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332936832/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018206.518761:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018206.518765:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018206.518770:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332936064:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018206.518778:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332936192:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018206.518797:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018206.518803:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018206/real 1734018206] req@ffff8800a705df80 x1818247332936192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018206.518815:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332936192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018206.518823:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018206.518826:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018206.518830:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332936192:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018206.518855:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332936320:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018206.518869:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018206.518875:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018206/real 1734018206] req@ffff8800a705dc00 x1818247332936320/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018206.518886:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332936320/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018206.518894:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018206.518896:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018206.518899:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332936320:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018206.518905:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332936448:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018206.518916:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018206.518923:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018206/real 1734018206] req@ffff8800a705f100 x1818247332936448/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018206.518932:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332936448/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018206.518940:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018206.518942:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018206.518946:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332936448:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018206.518952:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332936576:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018206.518968:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018206.518974:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018206/real 1734018206] req@ffff8800a705c000 x1818247332936576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018206.518985:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332936576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018206.518992:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018206.518995:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018206.518998:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332936576:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018206.519004:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332936704:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018206.519016:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018206.519022:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018206/real 1734018206] req@ffff8800a705e300 x1818247332936704/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018206.519032:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332936704/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018206.519039:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018206.519042:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018206.519045:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332936704:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018206.519051:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332936832:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018206.519063:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018206.519069:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018206/real 1734018206] req@ffff8800a705ed80 x1818247332936832/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018206.519085:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332936832/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018206.519092:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018206.519095:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018206.519098:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332936832:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018207.120569:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018208.122522:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018209.124519:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018210.126512:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018211.128550:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018211.526376:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018211.526386:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018211.526430:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018211.526441:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2474 00000100:00080000:0.0:1734018211.526451:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018211.526459:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018211.526496:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332936960/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018211.526529:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018211.526534:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018211.526536:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018211.526541:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2474 00000100:00080000:0.0:1734018211.526544:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018211.526548:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018211.526557:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018211.526568:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332937088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018211.526579:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018211.526583:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018211.526585:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018211.526588:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2474 00000100:00080000:0.0:1734018211.526591:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018211.526594:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018211.526599:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018211.526609:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332937216/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018211.526617:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018211.526620:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018211.526622:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018211.526626:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2474 00000100:00080000:0.0:1734018211.526628:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018211.526631:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018211.526635:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018211.526644:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332937344/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018211.526644:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332936960:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018211.526653:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018211.526656:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018211.526657:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018211.526661:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2474 00000100:00080000:0.0:1734018211.526664:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018211.526668:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018211.526672:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018211.526680:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332937472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018211.526688:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018211.526691:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018211.526692:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018211.526696:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2474 00000100:00080000:0.0:1734018211.526698:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018211.526702:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018211.526706:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018211.526712:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018211.526714:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332937600/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018211.526721:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018211.526725:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018211.526726:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018211.526729:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2474 00000100:00080000:0.0:1734018211.526732:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018211.526734:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018211/real 1734018211] req@ffff8800a705d180 x1818247332936960/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018211.526735:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018211.526739:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018211.526747:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332937728/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018211.526754:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332936960/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018211.526764:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018211.526768:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018211.526772:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332936960:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018211.526779:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332937088:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018211.526796:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018211.526803:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018211/real 1734018211] req@ffff8800a705c380 x1818247332937088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018211.526814:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332937088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018211.526821:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018211.526825:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018211.526829:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332937088:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018211.526835:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332937216:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018211.526849:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018211.526855:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018211/real 1734018211] req@ffff8800a705ed80 x1818247332937216/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018211.526865:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332937216/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018211.526874:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018211.526876:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018211.526879:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332937216:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018211.526904:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332937344:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018211.526916:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018211.526922:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018211/real 1734018211] req@ffff8800a705e300 x1818247332937344/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018211.526933:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332937344/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018211.526942:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018211.526944:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018211.526947:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332937344:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018211.526952:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332937472:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018211.526967:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018211.526973:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018211/real 1734018211] req@ffff8800a705c000 x1818247332937472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018211.526984:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332937472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018211.526991:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018211.526993:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018211.526996:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332937472:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018211.527001:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332937600:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018211.527012:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018211.527018:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018211/real 1734018211] req@ffff8800a705f100 x1818247332937600/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018211.527029:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332937600/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018211.527036:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018211.527038:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018211.527041:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332937600:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018211.527047:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332937728:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018211.527059:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018211.527065:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018211/real 1734018211] req@ffff8800a705dc00 x1818247332937728/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018211.527082:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332937728/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018211.527090:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018211.527092:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018211.527094:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332937728:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018212.130548:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018213.132553:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018214.134527:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018215.136562:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018216.138511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018216.534408:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018216.534417:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018216.534421:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018216.534433:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2479 00000100:00080000:0.0:1734018216.534442:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018216.534450:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018216.534485:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332937856/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018216.534519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018216.534523:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018216.534525:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018216.534530:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2479 00000100:00080000:0.0:1734018216.534533:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018216.534537:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018216.534547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018216.534558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332937984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018216.534568:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018216.534571:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018216.534573:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018216.534577:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2479 00000100:00080000:0.0:1734018216.534579:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018216.534583:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018216.534589:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018216.534597:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332938112/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018216.534606:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018216.534609:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018216.534610:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018216.534614:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2479 00000100:00080000:0.0:1734018216.534616:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018216.534619:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018216.534624:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018216.534630:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332937856:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018216.534632:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332938240/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018216.534641:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018216.534644:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018216.534646:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018216.534649:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2479 00000100:00080000:0.0:1734018216.534651:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018216.534655:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018216.534659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018216.534667:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332938368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018216.534675:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018216.534678:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018216.534680:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018216.534684:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2479 00000100:00080000:0.0:1734018216.534687:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018216.534690:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018216.534694:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018216.534697:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018216.534702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332938496/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018216.534710:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018216.534713:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018216.534714:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018216.534718:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2479 00000100:00100000:1.0:1734018216.534720:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018216/real 1734018216] req@ffff8800a705df80 x1818247332937856/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018216.534721:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018216.534724:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018216.534728:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018216.534737:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332938624/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018216.534740:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332937856/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018216.534751:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018216.534755:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018216.534759:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332937856:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018216.534767:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332937984:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018216.534784:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018216.534791:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018216/real 1734018216] req@ffff8800a705c700 x1818247332937984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018216.534802:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332937984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018216.534810:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018216.534813:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018216.534817:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332937984:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018216.534823:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332938112:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018216.534837:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018216.534843:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018216/real 1734018216] req@ffff8800a705d500 x1818247332938112/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018216.534853:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332938112/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018216.534860:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018216.534863:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018216.534866:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332938112:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018216.534890:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332938240:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018216.534903:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018216.534909:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018216/real 1734018216] req@ffff8800a705ea00 x1818247332938240/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018216.534920:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332938240/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018216.534928:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018216.534930:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018216.534933:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332938240:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018216.534938:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332938368:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018216.534953:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018216.534959:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018216/real 1734018216] req@ffff8800a705fb80 x1818247332938368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018216.534970:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332938368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018216.534977:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018216.534979:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018216.534982:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332938368:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018216.534987:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332938496:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018216.534999:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018216.535005:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018216/real 1734018216] req@ffff8800a705e680 x1818247332938496/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018216.535015:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332938496/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018216.535023:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018216.535025:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018216.535028:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332938496:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018216.535033:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332938624:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018216.535044:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018216.535050:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018216/real 1734018216] req@ffff8800a705ca80 x1818247332938624/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018216.535067:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332938624/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018216.535075:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018216.535077:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018216.535080:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332938624:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018217.140527:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018218.142540:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018219.144534:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018220.146473:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018221.148484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018221.542339:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018221.542349:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018221.542353:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018221.542364:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2484 00000100:00080000:0.0:1734018221.542374:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018221.542382:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018221.542417:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332938752/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018221.542459:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018221.542463:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018221.542465:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018221.542469:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2484 00000100:00080000:0.0:1734018221.542472:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018221.542476:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018221.542485:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018221.542496:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332938880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018221.542505:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018221.542509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018221.542510:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018221.542514:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2484 00000100:00080000:0.0:1734018221.542517:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018221.542520:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018221.542525:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018221.542535:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332939008/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018221.542543:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018221.542546:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018221.542548:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018221.542551:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2484 00000100:00080000:0.0:1734018221.542553:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018221.542557:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018221.542562:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018221.542564:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332938752:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018221.542570:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332939136/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018221.542579:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018221.542582:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018221.542584:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018221.542587:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2484 00000100:00080000:0.0:1734018221.542590:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018221.542593:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018221.542597:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018221.542604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332939264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018221.542612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018221.542615:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018221.542617:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018221.542620:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2484 00000100:00080000:0.0:1734018221.542623:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018221.542626:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018221.542630:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734018221.542631:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018221.542638:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332939392/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018221.542646:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018221.542649:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018221.542651:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018221.542652:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018221/real 1734018221] req@ffff8800a705d880 x1818247332938752/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018221.542655:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2484 00000100:00080000:0.0:1734018221.542657:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018221.542661:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018221.542665:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018221.542671:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332938752/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018221.542672:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332939520/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018221.542682:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018221.542685:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018221.542690:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332938752:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018221.542697:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332938880:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018221.542713:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018221.542720:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018221/real 1734018221] req@ffff8800a705f800 x1818247332938880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018221.542731:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332938880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018221.542739:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018221.542742:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018221.542746:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332938880:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018221.542752:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332939008:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018221.542765:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018221.542771:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018221/real 1734018221] req@ffff8800a705ca80 x1818247332939008/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018221.542781:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332939008/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018221.542789:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018221.542792:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018221.542795:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332939008:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018221.542820:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332939136:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018221.542832:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018221.542838:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018221/real 1734018221] req@ffff8800a705e680 x1818247332939136/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018221.542848:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332939136/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018221.542856:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018221.542858:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018221.542861:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332939136:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018221.542866:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332939264:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018221.542882:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018221.542888:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018221/real 1734018221] req@ffff8800a705fb80 x1818247332939264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018221.542897:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332939264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018221.542905:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018221.542907:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018221.542910:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332939264:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018221.542915:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332939392:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018221.542926:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018221.542932:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018221/real 1734018221] req@ffff8800a705ea00 x1818247332939392/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018221.542942:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332939392/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018221.542949:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018221.542951:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018221.542954:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332939392:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018221.542960:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332939520:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018221.542971:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018221.542977:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018221/real 1734018221] req@ffff8800a705d500 x1818247332939520/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018221.542993:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332939520/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018221.543000:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018221.543002:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018221.543005:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332939520:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018222.150518:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018223.152526:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018224.154489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018225.156521:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018226.158481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018226.550372:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018226.550385:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018226.550391:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018226.550406:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2489 00000100:00080000:0.0:1734018226.550419:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018226.550429:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018226.550478:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332939648/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018226.550517:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018226.550522:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018226.550524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018226.550529:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2489 00000100:00080000:0.0:1734018226.550533:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018226.550537:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018226.550549:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018226.550561:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332939776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018226.550573:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018226.550577:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018226.550579:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018226.550583:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2489 00000100:00080000:0.0:1734018226.550585:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018226.550589:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018226.550594:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018226.550603:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332939904/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018226.550612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018226.550615:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018226.550616:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018226.550620:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2489 00000100:00080000:0.0:1734018226.550622:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018226.550626:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018226.550630:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018226.550639:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332940032/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018226.550649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018226.550652:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018226.550653:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018226.550657:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2489 00000100:00100000:1.0:1734018226.550659:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332939648:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018226.550661:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018226.550664:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018226.550670:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018226.550679:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332940160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018226.550687:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018226.550690:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018226.550692:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018226.550695:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2489 00000100:00080000:0.0:1734018226.550698:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018226.550702:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018226.550706:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018226.550714:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332940288/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018226.550722:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018226.550725:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018226.550726:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018226.550730:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2489 00000800:00000100:1.0:1734018226.550731:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018226.550734:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018226.550737:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018226.550741:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018226.550752:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332940416/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018226.550756:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018226/real 1734018226] req@ffff8800a705c700 x1818247332939648/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018226.550777:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332939648/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018226.550789:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018226.550793:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018226.550798:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332939648:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018226.550806:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332939776:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018226.550824:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018226.550831:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018226/real 1734018226] req@ffff8800a705df80 x1818247332939776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018226.550842:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332939776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018226.550850:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018226.550854:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018226.550857:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332939776:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018226.550864:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332939904:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018226.550878:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018226.550884:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018226/real 1734018226] req@ffff8800a705dc00 x1818247332939904/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018226.550894:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332939904/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018226.550902:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018226.550905:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018226.550908:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332939904:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018226.550933:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332940032:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018226.550946:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018226.550952:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018226/real 1734018226] req@ffff8800a705f100 x1818247332940032/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018226.550963:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332940032/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018226.550971:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018226.550973:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018226.550976:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332940032:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018226.550982:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332940160:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018226.550998:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018226.551004:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018226/real 1734018226] req@ffff8800a705c000 x1818247332940160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018226.551032:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332940160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018226.551040:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018226.551043:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018226.551046:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332940160:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018226.551053:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332940288:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018226.551067:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018226.551073:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018226/real 1734018226] req@ffff8800a705e300 x1818247332940288/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018226.551090:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332940288/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018226.551097:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018226.551100:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018226.551103:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332940288:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018226.551109:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332940416:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018226.551121:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018226.551127:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018226/real 1734018226] req@ffff8800a705ed80 x1818247332940416/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018226.551168:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332940416/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018226.551177:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018226.551179:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018226.551182:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332940416:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018227.160478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018228.162451:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018229.164459:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018230.166476:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018231.168514:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018231.558369:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018231.558379:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018231.558383:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018231.558394:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2494 00000100:00080000:0.0:1734018231.558405:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018231.558413:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018231.558448:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332940544/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018231.558485:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018231.558489:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018231.558491:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018231.558496:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2494 00000100:00080000:0.0:1734018231.558499:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018231.558503:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018231.558513:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018231.558524:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332940672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018231.558535:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018231.558546:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018231.558548:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018231.558552:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2494 00000100:00080000:0.0:1734018231.558555:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018231.558558:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018231.558564:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018231.558574:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332940800/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018231.558583:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018231.558586:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018231.558588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018231.558591:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2494 00000100:00080000:0.0:1734018231.558594:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018231.558597:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018231.558601:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018231.558609:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332940928/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018231.558612:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332940544:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018231.558619:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018231.558622:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018231.558624:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018231.558627:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2494 00000100:00080000:0.0:1734018231.558629:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018231.558633:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018231.558637:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018231.558644:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332941056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018231.558652:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018231.558655:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018231.558658:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018231.558661:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2494 00000100:00080000:0.0:1734018231.558664:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018231.558667:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018231.558672:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018231.558678:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018231.558679:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332941184/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018231.558687:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018231.558690:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018231.558691:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018231.558695:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2494 00000100:00080000:0.0:1734018231.558697:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018231.558701:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018231.558702:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018231/real 1734018231] req@ffff8800a705c380 x1818247332940544/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018231.558706:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018231.558713:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332941312/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018231.558721:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332940544/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018231.558731:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018231.558735:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018231.558740:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332940544:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018231.558747:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332940672:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018231.558764:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018231.558771:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018231/real 1734018231] req@ffff8800a705d180 x1818247332940672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018231.558782:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332940672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018231.558790:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018231.558793:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018231.558796:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332940672:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018231.558802:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332940800:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018231.558816:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018231.558822:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018231/real 1734018231] req@ffff8800a705ed80 x1818247332940800/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018231.558832:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332940800/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018231.558839:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018231.558842:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018231.558845:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332940800:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018231.558869:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332940928:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018231.558881:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018231.558887:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018231/real 1734018231] req@ffff8800a705e300 x1818247332940928/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018231.558898:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332940928/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018231.558906:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018231.558908:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018231.558911:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332940928:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018231.558916:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332941056:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018231.558932:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018231.558938:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018231/real 1734018231] req@ffff8800a705c000 x1818247332941056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018231.558948:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332941056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018231.558956:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018231.558958:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018231.558961:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332941056:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018231.558966:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332941184:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018231.558977:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018231.558983:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018231/real 1734018231] req@ffff8800a705f100 x1818247332941184/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018231.558993:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332941184/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018231.559000:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018231.559003:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018231.559006:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332941184:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018231.559011:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332941312:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018231.559023:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018231.559029:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018231/real 1734018231] req@ffff8800a705dc00 x1818247332941312/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018231.559045:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332941312/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018231.559105:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018231.559107:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018231.559111:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332941312:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018232.170480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018233.172531:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018234.174478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018235.176515:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018236.178460:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018236.566331:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018236.566342:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018236.566347:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018236.566362:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2499 00000100:00080000:0.0:1734018236.566373:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018236.566382:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018236.566429:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332941440/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018236.566474:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018236.566479:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018236.566481:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018236.566486:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2499 00000100:00080000:0.0:1734018236.566490:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018236.566494:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018236.566507:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018236.566519:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332941568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018236.566528:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018236.566532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018236.566534:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018236.566538:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2499 00000100:00080000:0.0:1734018236.566541:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018236.566544:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018236.566550:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018236.566558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332941696/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018236.566567:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018236.566570:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018236.566571:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018236.566575:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2499 00000100:00080000:0.0:1734018236.566577:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018236.566581:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018236.566586:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018236.566594:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332941440:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018236.566595:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332941824/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018236.566605:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018236.566609:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018236.566610:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018236.566614:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2499 00000100:00080000:0.0:1734018236.566616:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018236.566620:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018236.566625:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018236.566633:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332941952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018236.566641:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018236.566644:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018236.566645:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018236.566650:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2499 00000100:00080000:0.0:1734018236.566656:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018236.566660:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018236.566664:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018236.566670:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018236.566673:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332942080/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018236.566681:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018236.566685:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018236.566687:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018236.566691:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2499 00000100:00080000:0.0:1734018236.566694:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018236.566694:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018236/real 1734018236] req@ffff8800a705df80 x1818247332941440/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018236.566698:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018236.566702:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018236.566713:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332942208/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018236.566714:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332941440/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018236.566727:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018236.566730:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018236.566736:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332941440:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018236.566744:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332941568:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018236.566763:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018236.566770:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018236/real 1734018236] req@ffff8800a705c700 x1818247332941568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018236.566781:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332941568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018236.566789:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018236.566793:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018236.566796:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332941568:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018236.566803:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332941696:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018236.566818:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018236.566824:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018236/real 1734018236] req@ffff8800a705d500 x1818247332941696/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018236.566834:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332941696/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018236.566842:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018236.566844:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018236.566848:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332941696:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018236.566874:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332941824:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018236.566887:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018236.566893:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018236/real 1734018236] req@ffff8800a705ea00 x1818247332941824/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018236.566904:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332941824/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018236.566912:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018236.566914:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018236.566917:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332941824:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018236.566923:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332941952:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018236.566939:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018236.566945:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018236/real 1734018236] req@ffff8800a705fb80 x1818247332941952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018236.566955:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332941952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018236.567008:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018236.567010:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018236.567014:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332941952:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018236.567021:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332942080:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018236.567038:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018236.567045:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018236/real 1734018236] req@ffff8800a705e680 x1818247332942080/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018236.567055:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332942080/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018236.567063:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018236.567065:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018236.567068:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332942080:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018236.567073:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332942208:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018236.567086:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018236.567092:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018236/real 1734018236] req@ffff8800a705ca80 x1818247332942208/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018236.567109:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332942208/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018236.567116:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018236.567119:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018236.567122:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332942208:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018237.180508:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018238.182513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018239.184477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018240.186465:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018241.188424:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018241.574348:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018241.574360:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018241.574364:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018241.574374:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2504 00000100:00080000:0.0:1734018241.574384:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018241.574392:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018241.574429:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332942336/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018241.574463:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018241.574470:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018241.574473:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018241.574480:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2504 00000100:00080000:0.0:1734018241.574485:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018241.574491:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018241.574504:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018241.574522:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332942464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018241.574527:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332942336:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018241.574539:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018241.574543:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018241.574545:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018241.574550:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2504 00000100:00080000:0.0:1734018241.574555:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018241.574559:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018241.574566:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018241.574580:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332942592/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018241.574592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018241.574597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1734018241.574598:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018241.574600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018241.574604:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2504 00000100:00080000:0.0:1734018241.574608:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018241.574613:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018241.574619:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018241.574622:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018241/real 1734018241] req@ffff8800a705f800 x1818247332942336/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018241.574632:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332942720/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018241.574645:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018241.574645:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332942336/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018241.574650:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018241.574653:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018241.574658:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2504 00000100:00080000:1.0:1734018241.574660:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018241.574663:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1734018241.574664:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:0.0:1734018241.574668:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018241.574670:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332942336:192.168.201.152@tcp:250:kworker.0 00000008:00100000:0.0:1734018241.574674:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018241.574686:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332942848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018241.574697:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332942464:192.168.201.152@tcp:38:kworker.0 00000100:00080000:0.0:1734018241.574699:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018241.574704:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018241.574706:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018241.574711:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2504 00000100:00080000:0.0:1734018241.574717:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018241.574722:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018241.574722:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734018241.574729:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018241.574734:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018241/real 1734018241] req@ffff8800a705d880 x1818247332942464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018241.574742:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332942976/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018241.574751:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332942464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018241.574756:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018241.574761:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018241.574764:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734018241.574764:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018241.574768:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2504 00000100:00080000:1.0:1734018241.574768:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00080000:0.0:1734018241.574772:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018241.574774:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332942464:192.168.201.152@tcp:38:kworker.0 00000100:00080000:0.0:1734018241.574777:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018241.574783:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332942592:192.168.201.152@tcp:8:kworker.0 00000008:00100000:0.0:1734018241.574784:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018241.574796:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332943104/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018241.574805:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018241.574815:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018241/real 1734018241] req@ffff8800a705ca80 x1818247332942592/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018241.574831:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332942592/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018241.574844:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018241.574847:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018241.574853:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332942592:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018241.574861:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332942720:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018241.574920:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018241.574933:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018241/real 1734018241] req@ffff8800a705e680 x1818247332942720/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018241.574945:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332942720/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018241.574954:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018241.574956:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018241.574960:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332942720:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018241.574977:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332942848:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018241.574998:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018241.575005:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018241/real 1734018241] req@ffff8800a705fb80 x1818247332942848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018241.575015:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332942848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018241.575023:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018241.575025:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018241.575028:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332942848:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018241.575033:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332942976:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018241.575045:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018241.575051:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018241/real 1734018241] req@ffff8800a705ea00 x1818247332942976/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018241.575061:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332942976/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018241.575068:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018241.575070:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018241.575073:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332942976:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018241.575078:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332943104:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018241.575089:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018241.575095:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018241/real 1734018241] req@ffff8800a705d500 x1818247332943104/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018241.575111:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332943104/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018241.575119:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018241.575121:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018241.575124:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332943104:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018242.190480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018243.192518:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018244.194479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018245.196513:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018246.198497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018246.582350:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018246.582361:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018246.582367:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018246.582381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2509 00000100:00080000:0.0:1734018246.582393:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018246.582402:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018246.582451:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332943232/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018246.582489:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018246.582494:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018246.582496:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018246.582501:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2509 00000100:00080000:0.0:1734018246.582505:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018246.582509:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018246.582521:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018246.582533:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332943360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018246.582544:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018246.582547:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018246.582549:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018246.582554:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2509 00000100:00080000:0.0:1734018246.582556:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018246.582560:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018246.582566:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018246.582575:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332943488/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018246.582584:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018246.582588:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018246.582589:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018246.582593:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2509 00000100:00080000:0.0:1734018246.582595:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018246.582599:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018246.582604:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018246.582612:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332943616/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018246.582622:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018246.582625:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018246.582627:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018246.582630:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2509 00000100:00080000:0.0:1734018246.582633:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018246.582636:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018246.582641:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734018246.582645:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332943232:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018246.582650:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332943744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018246.582659:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018246.582663:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018246.582664:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018246.582669:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2509 00000100:00080000:0.0:1734018246.582672:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018246.582675:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018246.582680:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018246.582688:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332943872/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018246.582696:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018246.582699:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018246.582701:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018246.582704:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2509 00000100:00080000:0.0:1734018246.582708:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018246.582711:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018246.582715:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018246.582722:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018246.582723:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332944000/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018246.582747:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018246/real 1734018246] req@ffff8800a705c700 x1818247332943232/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018246.582768:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332943232/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018246.582780:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018246.582783:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018246.582788:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332943232:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018246.582841:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332943360:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018246.582863:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018246.582872:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018246/real 1734018246] req@ffff8800a705df80 x1818247332943360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018246.582884:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332943360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018246.582892:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018246.582896:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018246.582899:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332943360:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018246.582906:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332943488:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018246.582921:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018246.582927:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018246/real 1734018246] req@ffff8800a705dc00 x1818247332943488/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018246.582937:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332943488/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018246.582945:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018246.582948:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018246.582951:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332943488:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018246.582977:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332943616:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018246.582990:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018246.582997:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018246/real 1734018246] req@ffff8800a705f100 x1818247332943616/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018246.583007:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332943616/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018246.583015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018246.583018:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018246.583021:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332943616:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018246.583027:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332943744:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018246.583042:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018246.583049:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018246/real 1734018246] req@ffff8800a705c000 x1818247332943744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018246.583059:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332943744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018246.583067:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018246.583069:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018246.583072:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332943744:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018246.583078:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332943872:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018246.583090:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018246.583096:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018246/real 1734018246] req@ffff8800a705e300 x1818247332943872/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018246.583106:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332943872/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018246.583113:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018246.583115:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018246.583118:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332943872:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018246.583124:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332944000:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018246.583137:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018246.583143:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018246/real 1734018246] req@ffff8800a705ed80 x1818247332944000/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018246.583159:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332944000/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018246.583167:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018246.583169:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018246.583172:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332944000:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018247.200526:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018248.202480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018249.204513:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018250.206485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018251.208487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018251.590341:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018251.590353:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018251.590359:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018251.590374:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2514 00000100:00080000:0.0:1734018251.590386:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018251.590395:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018251.590442:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332944128/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018251.590488:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018251.590493:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018251.590495:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018251.590500:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2514 00000100:00080000:0.0:1734018251.590504:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018251.590509:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018251.590520:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018251.590533:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332944256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018251.590543:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018251.590546:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018251.590548:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018251.590552:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2514 00000100:00080000:0.0:1734018251.590554:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018251.590558:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018251.590563:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018251.590574:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332944384/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018251.590583:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018251.590586:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018251.590588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018251.590591:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2514 00000100:00080000:0.0:1734018251.590594:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018251.590598:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018251.590603:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018251.590612:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332944512/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018251.590621:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018251.590625:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018251.590626:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018251.590630:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2514 00000100:00100000:1.0:1734018251.590631:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332944128:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018251.590633:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018251.590637:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018251.590641:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018251.590650:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332944640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018251.590659:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018251.590662:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018251.590664:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018251.590668:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2514 00000100:00080000:0.0:1734018251.590670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018251.590674:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018251.590679:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018251.590687:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332944768/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018251.590695:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018251.590700:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018251.590702:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018251.590747:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2514 00000100:00080000:0.0:1734018251.590752:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018251.590756:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018251.590761:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018251.590770:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332944256:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018251.590772:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332944896/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018251.590796:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332944384:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018251.590821:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332944512:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018251.590836:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332944640:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018251.590852:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332944768:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018251.590868:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332944896:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018252.210486:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018253.212485:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018254.214476:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1734018254.596459:0:1955:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:2.0:1734018254.596510:0:1955:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1734018254.596530:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018254.596565:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018254.596573:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018254.596581:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018254.596587:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018254.596593:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018254.596599:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734018254.596711:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018251/real 1734018254] req@ffff8800a705e300 x1818247332944512/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018254.596741:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332944512/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018254.596753:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018254.596757:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018254.596763:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332944512:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018254.596772:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018251/real 1734018254] req@ffff8800a705c000 x1818247332944640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018254.596792:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332944640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018254.596805:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018254.596809:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018254.596816:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332944640:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018254.596826:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018251/real 1734018254] req@ffff8800a705f100 x1818247332944768/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018254.596845:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332944768/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018254.596857:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018254.596861:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018254.596866:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332944768:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018254.596876:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018251/real 1734018254] req@ffff8800a705dc00 x1818247332944896/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018254.596894:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332944896/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018254.596906:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018254.596910:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018254.596915:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332944896:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018254.596929:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018251/real 1734018254] req@ffff8800a705d180 x1818247332944128/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018254.596948:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332944128/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018254.596961:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018254.596964:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018254.596969:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332944128:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018254.596978:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018251/real 1734018254] req@ffff8800a705c380 x1818247332944256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018254.596995:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332944256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018254.597008:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018254.597012:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018254.597018:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332944256:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018254.597028:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018251/real 1734018254] req@ffff8800a705ed80 x1818247332944384/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018254.597045:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332944384/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018254.597057:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018254.597061:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018254.597067:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332944384:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018255.216482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018256.218481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018256.598359:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018256.598368:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018256.598372:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018256.598383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2519 00000100:00080000:0.0:1734018256.598393:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018256.598401:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018256.598436:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332945024/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018256.598469:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018256.598473:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018256.598475:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018256.598480:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2519 00000100:00080000:0.0:1734018256.598483:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018256.598487:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018256.598497:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018256.598508:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332945152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018256.598519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018256.598522:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018256.598524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018256.598527:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2519 00000100:00080000:0.0:1734018256.598530:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018256.598533:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018256.598538:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018256.598545:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332945280/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018256.598554:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018256.598557:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018256.598559:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018256.598562:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2519 00000100:00080000:0.0:1734018256.598564:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018256.598568:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018256.598570:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332945024:192.168.201.152@tcp:250:kworker.0 00000008:00100000:0.0:1734018256.598573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018256.598580:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332945408/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018256.598589:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018256.598592:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018256.598594:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018256.598597:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2519 00000100:00080000:0.0:1734018256.598600:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018256.598603:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018256.598608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018256.598616:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332945536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018256.598653:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018256.598656:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018256.598658:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018256.598662:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2519 00000100:00080000:0.0:1734018256.598665:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018256.598669:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018256.598673:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734018256.598674:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018256.598682:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332945664/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018256.598691:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018256.598694:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018256.598696:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018256.598696:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018256/real 1734018256] req@ffff8800a705df80 x1818247332945024/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018256.598699:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2519 00000100:00080000:0.0:1734018256.598702:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018256.598705:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018256.598709:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018256.598716:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332945024/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018256.598718:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332945792/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018256.598727:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018256.598730:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018256.598735:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332945024:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018256.598743:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332945152:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018256.598761:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018256.598768:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018256/real 1734018256] req@ffff8800a705c700 x1818247332945152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018256.598779:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332945152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018256.598787:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018256.598790:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018256.598793:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332945152:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018256.598799:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332945280:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018256.598812:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018256.598818:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018256/real 1734018256] req@ffff8800a705d500 x1818247332945280/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018256.598828:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332945280/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018256.598837:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018256.598839:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018256.598842:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332945280:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018256.598866:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332945408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018256.598879:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018256.598885:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018256/real 1734018256] req@ffff8800a705ea00 x1818247332945408/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018256.598895:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332945408/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018256.598903:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018256.598905:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018256.598908:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332945408:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018256.598913:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332945536:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018256.598925:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018256.598931:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018256/real 1734018256] req@ffff8800a705fb80 x1818247332945536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018256.598940:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332945536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018256.598948:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018256.598950:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018256.598953:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332945536:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018256.598959:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332945664:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018256.598971:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018256.598977:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018256/real 1734018256] req@ffff8800a705e680 x1818247332945664/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018256.598987:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332945664/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018256.598994:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018256.598997:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018256.599000:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332945664:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018256.599005:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332945792:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018256.599017:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018256.599022:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018256/real 1734018256] req@ffff8800a705ca80 x1818247332945792/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018256.599032:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332945792/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018256.599039:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018256.599041:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018256.599044:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332945792:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018257.220480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018258.222420:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018259.224528:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018260.226481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018261.228521:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018261.606361:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018261.606371:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018261.606376:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018261.606386:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2524 00000100:00080000:0.0:1734018261.606396:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018261.606404:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018261.606441:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332945920/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018261.606474:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018261.606478:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018261.606481:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018261.606485:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2524 00000100:00080000:0.0:1734018261.606488:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018261.606492:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018261.606502:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018261.606513:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332946048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018261.606523:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018261.606526:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018261.606528:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018261.606532:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2524 00000100:00080000:0.0:1734018261.606567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018261.606572:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018261.606579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018261.606590:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332946176/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018261.606600:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018261.606603:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018261.606605:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018261.606608:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2524 00000100:00080000:0.0:1734018261.606611:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018261.606614:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018261.606617:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332945920:192.168.201.152@tcp:250:kworker.0 00000008:00100000:0.0:1734018261.606619:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018261.606626:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332946304/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018261.606635:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018261.606639:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018261.606640:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018261.606644:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2524 00000100:00080000:0.0:1734018261.606646:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018261.606650:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018261.606654:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018261.606662:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332946432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018261.606670:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018261.606673:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018261.606675:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018261.606679:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2524 00000100:00080000:0.0:1734018261.606682:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018261.606684:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018261.606685:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018261.606690:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018261.606697:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332946560/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018261.606705:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018261.606706:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018261/real 1734018261] req@ffff8800a705d880 x1818247332945920/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018261.606708:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018261.606710:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018261.606713:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2524 00000100:00080000:0.0:1734018261.606716:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018261.606719:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018261.606723:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018261.606727:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332945920/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018261.606731:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332946688/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018261.606738:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018261.606742:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018261.606747:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332945920:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018261.606754:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332946048:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018261.606771:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018261.606778:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018261/real 1734018261] req@ffff8800a705f800 x1818247332946048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018261.606790:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332946048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018261.606797:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018261.606800:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018261.606803:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332946048:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018261.606809:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332946176:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018261.606822:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018261.606828:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018261/real 1734018261] req@ffff8800a705ca80 x1818247332946176/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018261.606838:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332946176/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018261.606846:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018261.606849:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018261.606852:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332946176:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018261.606877:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332946304:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018261.606890:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018261.606896:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018261/real 1734018261] req@ffff8800a705e680 x1818247332946304/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018261.606907:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332946304/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018261.606914:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018261.606916:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018261.606919:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332946304:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018261.606924:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332946432:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018261.606936:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018261.606942:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018261/real 1734018261] req@ffff8800a705fb80 x1818247332946432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018261.606952:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332946432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018261.606959:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018261.606961:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018261.606964:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332946432:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018261.606969:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332946560:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018261.606985:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018261.606991:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018261/real 1734018261] req@ffff8800a705ea00 x1818247332946560/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018261.607001:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332946560/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018261.607008:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018261.607010:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018261.607013:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332946560:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018261.607018:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332946688:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018261.607030:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018261.607036:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018261/real 1734018261] req@ffff8800a705d500 x1818247332946688/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018261.607046:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332946688/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018261.607054:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018261.607056:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018261.607059:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332946688:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018262.230494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018263.232470:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018264.234477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018265.236519:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018266.238447:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018266.614353:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018266.614364:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018266.614369:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018266.614382:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2529 00000100:00080000:0.0:1734018266.614394:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018266.614403:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018266.614488:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332946816/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018266.614533:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018266.614538:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018266.614540:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018266.614545:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2529 00000100:00080000:0.0:1734018266.614549:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018266.614554:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018266.614565:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018266.614579:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332946944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018266.614590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018266.614593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018266.614595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018266.614599:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2529 00000100:00080000:0.0:1734018266.614602:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018266.614605:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018266.614610:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018266.614619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332947072/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018266.614628:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018266.614631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018266.614633:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018266.614636:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2529 00000100:00080000:0.0:1734018266.614639:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018266.614643:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018266.614648:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018266.614656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332947200/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018266.614666:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018266.614669:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018266.614671:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018266.614674:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332946816:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018266.614675:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2529 00000100:00080000:0.0:1734018266.614678:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018266.614681:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018266.614686:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018266.614695:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332947328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018266.614704:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018266.614707:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018266.614709:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018266.614713:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2529 00000100:00080000:0.0:1734018266.614716:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018266.614719:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018266.614724:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018266.614732:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332947456/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018266.614740:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018266.614744:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018266.614746:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018266.614750:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2529 00000800:00000100:1.0:1734018266.614751:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018266.614754:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018266.614757:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018266.614761:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018266.614771:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332947584/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018266.614776:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018266/real 1734018266] req@ffff8800a705c700 x1818247332946816/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018266.614804:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332946816/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018266.614816:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018266.614819:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018266.614824:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332946816:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018266.614832:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332946944:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018266.614850:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018266.614857:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018266/real 1734018266] req@ffff8800a705df80 x1818247332946944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018266.614868:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332946944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018266.614876:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018266.614879:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018266.614882:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332946944:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018266.614889:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332947072:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018266.614903:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018266.614909:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018266/real 1734018266] req@ffff8800a705ed80 x1818247332947072/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018266.614919:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332947072/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018266.614926:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018266.614930:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018266.614933:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332947072:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018266.614960:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332947200:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018266.614973:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018266.614980:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018266/real 1734018266] req@ffff8800a705c380 x1818247332947200/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018266.614990:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332947200/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018266.614998:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018266.615000:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018266.615003:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332947200:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018266.615009:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332947328:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018266.615022:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018266.615028:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018266/real 1734018266] req@ffff8800a705d180 x1818247332947328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018266.615038:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332947328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018266.615046:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018266.615049:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018266.615051:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332947328:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018266.615057:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332947456:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018266.615073:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018266.615079:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018266/real 1734018266] req@ffff8800a705dc00 x1818247332947456/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018266.615090:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332947456/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018266.615097:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018266.615099:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018266.615102:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332947456:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018266.615108:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332947584:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018266.615120:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018266.615126:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018266/real 1734018266] req@ffff8800a705f100 x1818247332947584/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018266.615136:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332947584/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018266.615144:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018266.615146:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018266.615149:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332947584:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018267.240497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018268.242482:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018269.244528:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018270.246489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018271.248526:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018271.622398:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018271.622410:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018271.622416:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018271.622430:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2534 00000100:00080000:0.0:1734018271.622442:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018271.622452:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018271.622500:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332947712/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018271.622539:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018271.622543:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018271.622546:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018271.622551:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2534 00000100:00080000:0.0:1734018271.622555:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018271.622559:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018271.622571:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018271.622583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332947840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018271.622594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018271.622598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018271.622600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018271.622603:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2534 00000100:00080000:0.0:1734018271.622606:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018271.622610:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018271.622615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018271.622627:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332947968/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018271.622636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018271.622639:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018271.622640:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018271.622644:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2534 00000100:00080000:0.0:1734018271.622647:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018271.622650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018271.622655:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018271.622665:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332948096/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018271.622666:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332947712:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018271.622675:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018271.622679:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018271.622680:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018271.622684:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2534 00000100:00080000:0.0:1734018271.622687:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018271.622691:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018271.622696:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018271.622705:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332948224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018271.622713:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018271.622716:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018271.622718:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018271.622722:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2534 00000100:00080000:0.0:1734018271.622734:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018271.622738:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018271.622743:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018271.622745:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018271.622752:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332948352/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018271.622761:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018271.622764:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018271.622766:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018271.622770:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2534 00000100:00100000:1.0:1734018271.622771:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018271/real 1734018271] req@ffff8800a705c000 x1818247332947712/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018271.622773:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018271.622777:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018271.622781:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018271.622790:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332948480/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018271.622798:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332947712/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018271.622810:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018271.622814:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018271.622819:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332947712:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018271.622827:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332947840:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018271.622846:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018271.622853:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018271/real 1734018271] req@ffff8800a705e300 x1818247332947840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018271.622865:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332947840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018271.622873:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018271.622875:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018271.622879:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332947840:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018271.622886:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332947968:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018271.622900:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018271.622906:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018271/real 1734018271] req@ffff8800a705f100 x1818247332947968/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018271.622916:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332947968/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018271.622924:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018271.622927:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018271.622930:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332947968:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018271.622957:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332948096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018271.622971:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018271.622977:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018271/real 1734018271] req@ffff8800a705dc00 x1818247332948096/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018271.622987:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332948096/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018271.622995:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018271.622997:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018271.623000:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332948096:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018271.623006:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332948224:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018271.623019:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018271.623025:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018271/real 1734018271] req@ffff8800a705d180 x1818247332948224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018271.623035:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332948224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018271.623042:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018271.623044:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018271.623047:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332948224:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018271.623054:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332948352:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018271.623070:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018271.623076:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018271/real 1734018271] req@ffff8800a705c380 x1818247332948352/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018271.623087:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332948352/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018271.623093:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018271.623096:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018271.623099:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332948352:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018271.623104:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332948480:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018271.623116:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018271.623122:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018271/real 1734018271] req@ffff8800a705ed80 x1818247332948480/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018271.623132:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332948480/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018271.623139:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018271.623141:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018271.623144:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332948480:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018272.250480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018273.252474:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018274.254557:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018275.256500:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018276.258514:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018276.630396:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018276.630408:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018276.630414:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018276.630429:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2539 00000100:00080000:0.0:1734018276.630441:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018276.630450:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018276.630500:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332948608/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018276.630538:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018276.630543:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018276.630545:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018276.630551:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2539 00000100:00080000:0.0:1734018276.630554:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018276.630558:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018276.630570:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018276.630582:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332948736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018276.630593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018276.630596:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018276.630598:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018276.630602:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2539 00000100:00080000:0.0:1734018276.630605:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018276.630608:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018276.630614:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018276.630624:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332948864/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018276.630633:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018276.630636:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018276.630638:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018276.630641:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2539 00000100:00080000:0.0:1734018276.630644:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018276.630647:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018276.630653:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018276.630662:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332948992/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018276.630663:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332948608:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018276.630671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018276.630675:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018276.630676:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018276.630680:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2539 00000100:00080000:0.0:1734018276.630683:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018276.630686:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018276.630691:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018276.630700:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332949120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018276.630710:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018276.630713:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018276.630715:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018276.630720:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2539 00000100:00080000:0.0:1734018276.630723:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018276.630727:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018276.630732:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018276.630738:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018276.630740:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332949248/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018276.630748:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018276.630751:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018276.630753:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018276.630756:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2539 00000100:00080000:0.0:1734018276.630759:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018276.630763:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018276.630763:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018276/real 1734018276] req@ffff8800a705df80 x1818247332948608/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018276.630767:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018276.630778:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332949376/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018276.630792:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332948608/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018276.630803:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018276.630806:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018276.630811:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332948608:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018276.630820:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332948736:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018276.630838:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018276.630845:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018276/real 1734018276] req@ffff8800a705c700 x1818247332948736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018276.630857:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332948736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018276.630865:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018276.630868:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018276.630871:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332948736:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018276.630878:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332948864:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018276.630892:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018276.630898:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018276/real 1734018276] req@ffff8800a705d500 x1818247332948864/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018276.630908:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332948864/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018276.630916:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018276.630919:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018276.630922:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332948864:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018276.630950:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332948992:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018276.630964:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018276.630970:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018276/real 1734018276] req@ffff8800a705ea00 x1818247332948992/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018276.630980:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332948992/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018276.630988:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018276.630990:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018276.630993:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332948992:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018276.630999:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332949120:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018276.631011:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018276.631017:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018276/real 1734018276] req@ffff8800a705fb80 x1818247332949120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018276.631027:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332949120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018276.631034:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018276.631037:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018276.631040:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332949120:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018276.631047:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332949248:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018276.631062:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018276.631069:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018276/real 1734018276] req@ffff8800a705e680 x1818247332949248/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018276.631079:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332949248/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018276.631086:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018276.631088:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018276.631091:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332949248:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018276.631097:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332949376:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018276.631109:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018276.631115:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018276/real 1734018276] req@ffff8800a705ca80 x1818247332949376/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018276.631124:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332949376/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018276.631132:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018276.631134:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018276.631137:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332949376:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018277.260451:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018278.262607:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018279.264519:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018280.266522:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018281.268495:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018281.638359:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018281.638370:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018281.638374:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018281.638389:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2544 00000100:00080000:0.0:1734018281.638401:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018281.638410:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018281.638459:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332949504/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018281.638506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018281.638510:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018281.638512:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018281.638517:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2544 00000100:00080000:0.0:1734018281.638521:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018281.638525:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018281.638537:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018281.638550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332949632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018281.638560:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018281.638564:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018281.638565:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018281.638569:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2544 00000100:00080000:0.0:1734018281.638572:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018281.638575:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018281.638581:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018281.638592:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332949760/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018281.638600:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018281.638604:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018281.638605:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018281.638609:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2544 00000100:00080000:0.0:1734018281.638611:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018281.638616:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018281.638621:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018281.638627:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332949504:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018281.638631:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332949888/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018281.638641:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018281.638644:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018281.638645:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018281.638649:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2544 00000100:00080000:0.0:1734018281.638653:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018281.638657:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018281.638662:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018281.638670:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332950016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018281.638678:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018281.638682:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018281.638684:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018281.638688:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2544 00000100:00080000:0.0:1734018281.638692:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018281.638695:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018281.638700:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018281.638704:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018281.638708:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332950144/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018281.638716:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018281.638719:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018281.638721:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018281.638725:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2544 00000100:00100000:1.0:1734018281.638728:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018281/real 1734018281] req@ffff8800a705f800 x1818247332949504/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018281.638729:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018281.638732:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018281.638737:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018281.638746:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332950272/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018281.638756:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332949504/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018281.638767:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018281.638771:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018281.638776:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332949504:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018281.638784:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332949632:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018281.638803:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018281.638810:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018281/real 1734018281] req@ffff8800a705d880 x1818247332949632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018281.638821:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332949632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018281.638829:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018281.638832:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018281.638835:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332949632:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018281.638841:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332949760:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018281.638856:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018281.638862:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018281/real 1734018281] req@ffff8800a705ca80 x1818247332949760/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018281.638872:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332949760/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018281.638880:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018281.638883:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018281.638886:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332949760:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018281.638912:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332949888:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018281.638926:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018281.638932:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018281/real 1734018281] req@ffff8800a705e680 x1818247332949888/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018281.638943:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332949888/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018281.638950:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018281.638952:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018281.638955:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332949888:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018281.638961:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332950016:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018281.638974:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018281.638980:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018281/real 1734018281] req@ffff8800a705fb80 x1818247332950016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018281.638990:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332950016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018281.638997:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018281.638999:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018281.639002:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332950016:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018281.639009:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332950144:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018281.639024:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018281.639031:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018281/real 1734018281] req@ffff8800a705ea00 x1818247332950144/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018281.639041:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332950144/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018281.639049:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018281.639051:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018281.639054:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332950144:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018281.639060:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332950272:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018281.639072:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018281.639078:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018281/real 1734018281] req@ffff8800a705d500 x1818247332950272/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018281.639087:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332950272/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018281.639094:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018281.639097:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018281.639100:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332950272:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018282.270515:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018283.272456:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018284.274517:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018285.276475:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018286.278492:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018286.646328:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018286.646342:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018286.646348:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018286.646362:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2549 00000100:00080000:0.0:1734018286.646374:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018286.646384:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018286.646433:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332950400/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018286.646472:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018286.646477:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018286.646479:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018286.646484:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2549 00000100:00080000:0.0:1734018286.646487:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018286.646492:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018286.646504:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018286.646516:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332950528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018286.646527:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018286.646531:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018286.646532:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018286.646536:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2549 00000100:00080000:0.0:1734018286.646539:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018286.646542:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018286.646547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018286.646556:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332950656/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018286.646565:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018286.646568:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018286.646570:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018286.646573:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2549 00000100:00080000:0.0:1734018286.646576:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018286.646579:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018286.646584:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018286.646593:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332950784/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018286.646603:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018286.646606:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1734018286.646607:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332950400:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018286.646608:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018286.646612:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2549 00000100:00080000:0.0:1734018286.646616:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018286.646620:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018286.646625:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018286.646634:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332950912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018286.646642:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018286.646646:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018286.646647:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018286.646651:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2549 00000100:00080000:0.0:1734018286.646670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018286.646673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018286.646678:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018286.646684:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018286.646687:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332951040/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018286.646695:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018286.646698:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018286.646700:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018286.646703:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2549 00000100:00080000:0.0:1734018286.646707:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018286.646708:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018286/real 1734018286] req@ffff8800a705c700 x1818247332950400/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018286.646711:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018286.646715:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018286.646725:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332951168/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018286.646737:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332950400/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018286.646748:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018286.646752:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018286.646757:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332950400:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018286.646765:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332950528:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018286.646783:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018286.646790:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018286/real 1734018286] req@ffff8800a705df80 x1818247332950528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018286.646801:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332950528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018286.646809:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018286.646812:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018286.646815:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332950528:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018286.646822:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332950656:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018286.646836:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018286.646842:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018286/real 1734018286] req@ffff8800a705ed80 x1818247332950656/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018286.646852:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332950656/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018286.646859:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018286.646862:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018286.646865:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332950656:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018286.646893:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332950784:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018286.646907:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018286.646913:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018286/real 1734018286] req@ffff8800a705c380 x1818247332950784/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018286.646923:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332950784/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018286.646930:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018286.646933:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018286.646936:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332950784:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018286.646942:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332950912:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018286.646954:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018286.646960:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018286/real 1734018286] req@ffff8800a705d180 x1818247332950912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018286.646970:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332950912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018286.646977:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018286.646979:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018286.646982:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332950912:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018286.646989:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332951040:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018286.647005:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018286.647011:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018286/real 1734018286] req@ffff8800a705dc00 x1818247332951040/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018286.647021:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332951040/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018286.647029:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018286.647031:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018286.647034:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332951040:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018286.647040:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332951168:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018286.647052:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018286.647057:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018286/real 1734018286] req@ffff8800a705f100 x1818247332951168/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018286.647067:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332951168/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018286.647074:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018286.647077:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018286.647080:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332951168:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018287.280480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018288.282547:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018289.284528:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018290.286526:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018291.288504:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018291.654358:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018291.654369:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018291.654374:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018291.654388:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2554 00000100:00080000:0.0:1734018291.654399:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018291.654409:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018291.654457:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332951296/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018291.654497:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018291.654502:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018291.654504:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018291.654509:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2554 00000100:00080000:0.0:1734018291.654513:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018291.654517:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018291.654530:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018291.654542:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332951424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018291.654553:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018291.654556:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018291.654558:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018291.654562:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2554 00000100:00080000:0.0:1734018291.654565:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018291.654568:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018291.654574:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018291.654586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332951552/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018291.654595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018291.654598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018291.654600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018291.654603:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2554 00000100:00080000:0.0:1734018291.654606:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018291.654609:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018291.654614:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018291.654624:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332951680/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018291.654624:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332951296:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018291.654633:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018291.654637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018291.654638:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018291.654643:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2554 00000100:00080000:0.0:1734018291.654646:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018291.654649:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018291.654654:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018291.654663:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332951808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018291.654671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018291.654674:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018291.654676:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018291.654681:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2554 00000100:00080000:0.0:1734018291.654686:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018291.654690:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018291.654694:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018291.654700:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018291.654703:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332951936/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018291.654710:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018291.654714:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018291.654715:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018291.654719:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2554 00000100:00080000:0.0:1734018291.654722:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018291.654725:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018291.654725:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018291/real 1734018291] req@ffff8800a705e300 x1818247332951296/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018291.654730:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018291.654739:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332952064/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018291.654752:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332951296/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018291.654764:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018291.654768:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018291.654773:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332951296:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018291.654781:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332951424:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018291.654799:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018291.654806:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018291/real 1734018291] req@ffff8800a705c000 x1818247332951424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018291.654817:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332951424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018291.654825:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018291.654828:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018291.654831:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332951424:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018291.654837:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332951552:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018291.654851:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018291.654857:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018291/real 1734018291] req@ffff8800a705f100 x1818247332951552/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018291.654867:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332951552/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018291.654874:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018291.654877:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018291.654881:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332951552:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018291.654907:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332951680:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018291.654920:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018291.654926:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018291/real 1734018291] req@ffff8800a705dc00 x1818247332951680/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018291.654937:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332951680/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018291.654944:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018291.654947:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018291.654950:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332951680:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018291.654955:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332951808:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018291.654967:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018291.654973:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018291/real 1734018291] req@ffff8800a705d180 x1818247332951808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018291.654983:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332951808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018291.654991:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018291.654993:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018291.654996:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332951808:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018291.655003:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332951936:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018291.655018:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018291.655057:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018291/real 1734018291] req@ffff8800a705c380 x1818247332951936/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018291.655069:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332951936/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018291.655077:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018291.655079:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018291.655083:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332951936:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018291.655089:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332952064:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018291.655105:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018291.655111:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018291/real 1734018291] req@ffff8800a705ed80 x1818247332952064/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018291.655121:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332952064/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018291.655129:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018291.655131:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018291.655134:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332952064:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018292.290500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018293.292474:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018294.294472:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018295.296499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018296.298465:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018296.662372:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018296.662386:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018296.662391:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018296.662406:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2559 00000100:00080000:0.0:1734018296.662418:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018296.662428:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018296.662477:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332952192/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018296.662525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018296.662530:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018296.662532:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018296.662538:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2559 00000100:00080000:0.0:1734018296.662541:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018296.662546:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018296.662557:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018296.662568:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332952320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018296.662578:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018296.662582:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018296.662584:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018296.662588:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2559 00000100:00080000:0.0:1734018296.662590:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018296.662594:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018296.662599:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018296.662608:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332952448/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018296.662616:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018296.662620:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018296.662621:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018296.662625:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2559 00000100:00080000:0.0:1734018296.662627:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018296.662632:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018296.662636:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018296.662646:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332952576/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018296.662654:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332952192:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018296.662655:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018296.662659:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018296.662660:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018296.662664:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2559 00000100:00080000:0.0:1734018296.662667:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018296.662670:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018296.662675:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018296.662684:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332952704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018296.662692:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018296.662696:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018296.662697:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018296.662702:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2559 00000100:00080000:0.0:1734018296.662715:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018296.662718:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018296.662724:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018296.662724:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018296.662732:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332952832/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018296.662741:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018296.662745:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018296.662747:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018296.662749:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018296/real 1734018296] req@ffff8800a705df80 x1818247332952192/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018296.662751:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2559 00000100:00080000:0.0:1734018296.662754:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018296.662758:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018296.662763:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018296.662773:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332952960/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018296.662776:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332952192/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018296.662787:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018296.662790:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018296.662795:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332952192:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018296.662804:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332952320:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018296.662822:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018296.662829:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018296/real 1734018296] req@ffff8800a705c700 x1818247332952320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018296.662840:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332952320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018296.662848:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018296.662850:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018296.662854:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332952320:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018296.662860:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332952448:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018296.662874:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018296.662880:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018296/real 1734018296] req@ffff8800a705d500 x1818247332952448/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018296.662889:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332952448/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018296.662897:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018296.662901:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018296.662904:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332952448:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018296.662930:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332952576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018296.662991:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018296.662999:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018296/real 1734018296] req@ffff8800a705ea00 x1818247332952576/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018296.663010:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332952576/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018296.663018:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018296.663021:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018296.663024:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332952576:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018296.663030:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332952704:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018296.663044:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018296.663050:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018296/real 1734018296] req@ffff8800a705fb80 x1818247332952704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018296.663061:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332952704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018296.663069:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018296.663071:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018296.663074:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332952704:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018296.663080:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332952832:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018296.663095:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018296.663102:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018296/real 1734018296] req@ffff8800a705e680 x1818247332952832/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018296.663112:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332952832/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018296.663120:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018296.663122:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018296.663125:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332952832:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018296.663131:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332952960:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018296.663142:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018296.663148:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018296/real 1734018296] req@ffff8800a705ca80 x1818247332952960/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018296.663158:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332952960/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018296.663165:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018296.663168:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018296.663170:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332952960:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018297.300509:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018298.302492:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018299.304515:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018300.306494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018301.308423:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018301.670394:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018301.670406:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018301.670411:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018301.670421:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2564 00000100:00080000:0.0:1734018301.670432:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018301.670440:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018301.670476:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332953088/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018301.670510:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018301.670514:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018301.670516:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018301.670521:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2564 00000100:00080000:0.0:1734018301.670524:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018301.670528:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018301.670538:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018301.670549:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332953216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018301.670560:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018301.670563:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018301.670565:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018301.670569:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2564 00000100:00080000:0.0:1734018301.670571:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018301.670575:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018301.670579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018301.670588:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332953344/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018301.670597:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018301.670600:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018301.670601:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018301.670605:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2564 00000100:00080000:0.0:1734018301.670607:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018301.670610:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018301.670615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018301.670618:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332953088:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018301.670623:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332953472/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018301.670632:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018301.670635:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018301.670637:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018301.670640:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2564 00000100:00080000:0.0:1734018301.670643:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018301.670647:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018301.670651:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018301.670658:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332953600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018301.670666:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018301.670669:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018301.670671:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018301.670674:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2564 00000100:00080000:0.0:1734018301.670677:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018301.670680:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018301.670684:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018301.670687:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018301.670691:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332953728/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018301.670699:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018301.670702:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018301.670704:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018301.670707:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2564 00000100:00080000:0.0:1734018301.670709:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018301.670709:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018301/real 1734018301] req@ffff8800a705d880 x1818247332953088/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018301.670713:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018301.670717:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018301.670725:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332953856/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018301.670737:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332953088/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018301.670746:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018301.670750:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018301.670755:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332953088:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018301.670762:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332953216:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018301.670779:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018301.670786:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018301/real 1734018301] req@ffff8800a705f800 x1818247332953216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018301.670797:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332953216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018301.670805:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018301.670807:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018301.670810:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332953216:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018301.670816:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332953344:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018301.670829:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018301.670835:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018301/real 1734018301] req@ffff8800a705ca80 x1818247332953344/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018301.670845:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332953344/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018301.670909:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018301.670914:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018301.670918:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332953344:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018301.670944:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332953472:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018301.670961:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018301.670968:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018301/real 1734018301] req@ffff8800a705e680 x1818247332953472/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018301.670979:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332953472/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018301.670987:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018301.670989:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018301.670992:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332953472:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018301.670997:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332953600:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018301.671009:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018301.671015:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018301/real 1734018301] req@ffff8800a705fb80 x1818247332953600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018301.671025:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332953600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018301.671033:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018301.671035:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018301.671038:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332953600:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018301.671043:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332953728:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018301.671058:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018301.671064:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018301/real 1734018301] req@ffff8800a705ea00 x1818247332953728/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018301.671075:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332953728/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018301.671082:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018301.671084:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018301.671087:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332953728:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018301.671092:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332953856:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018301.671103:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018301.671109:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018301/real 1734018301] req@ffff8800a705d500 x1818247332953856/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018301.671119:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332953856/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018301.671126:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018301.671128:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018301.671131:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332953856:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018302.310511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018303.312485:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018304.314522:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018305.316465:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018306.318469:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018306.678371:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018306.678384:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018306.678390:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018306.678404:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2569 00000100:00080000:0.0:1734018306.678416:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018306.678427:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018306.678476:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332953984/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018306.678514:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018306.678519:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018306.678521:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018306.678527:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2569 00000100:00080000:0.0:1734018306.678530:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018306.678535:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018306.678547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018306.678559:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332954112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018306.678570:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018306.678573:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018306.678575:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018306.678580:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2569 00000100:00080000:0.0:1734018306.678583:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018306.678587:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018306.678593:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018306.678603:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332954240/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018306.678614:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018306.678617:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018306.678618:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018306.678622:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2569 00000100:00080000:0.0:1734018306.678625:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018306.678628:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018306.678631:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332953984:192.168.201.152@tcp:250:kworker.0 00000008:00100000:0.0:1734018306.678633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018306.678641:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332954368/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018306.678649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018306.678653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018306.678655:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018306.678658:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2569 00000100:00080000:0.0:1734018306.678661:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018306.678665:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018306.678670:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018306.678679:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332954496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018306.678687:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018306.678690:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018306.678692:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018306.678696:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2569 00000100:00080000:0.0:1734018306.678700:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018306.678703:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018306.678704:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018306.678708:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018306.678716:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332954624/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018306.678725:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018306.678727:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018306/real 1734018306] req@ffff8800a705c700 x1818247332953984/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018306.678728:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018306.678730:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018306.678734:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2569 00000100:00080000:0.0:1734018306.678737:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018306.678740:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018306.678745:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018306.678754:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332953984/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018306.678755:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332954752/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018306.678766:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018306.678798:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018306.678804:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332953984:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018306.678813:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332954112:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018306.678835:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018306.678843:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018306/real 1734018306] req@ffff8800a705df80 x1818247332954112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018306.678854:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332954112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018306.678863:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018306.678866:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018306.678869:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332954112:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018306.678893:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332954240:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018306.678907:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018306.678914:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018306/real 1734018306] req@ffff8800a705ed80 x1818247332954240/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018306.678924:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332954240/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018306.678932:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018306.678935:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018306.678938:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332954240:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018306.678944:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332954368:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018306.678957:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018306.678962:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018306/real 1734018306] req@ffff8800a705c380 x1818247332954368/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018306.678972:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332954368/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018306.678979:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018306.678982:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018306.678985:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332954368:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018306.678990:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332954496:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018306.679003:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018306.679009:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018306/real 1734018306] req@ffff8800a705d180 x1818247332954496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018306.679019:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332954496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018306.679027:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018306.679029:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018306.679032:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332954496:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018306.679038:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332954624:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018306.679054:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018306.679060:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018306/real 1734018306] req@ffff8800a705dc00 x1818247332954624/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018306.679070:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332954624/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018306.679077:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018306.679079:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018306.679082:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332954624:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018306.679087:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332954752:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018306.679100:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018306.679105:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018306/real 1734018306] req@ffff8800a705f100 x1818247332954752/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018306.679115:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332954752/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018306.679122:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018306.679124:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018306.679127:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332954752:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018307.320484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018308.322564:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018309.324533:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018310.326474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018311.328466:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018311.686368:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018311.686381:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018311.686386:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018311.686401:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2574 00000100:00080000:0.0:1734018311.686412:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018311.686422:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018311.686470:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332954880/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018311.686516:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018311.686521:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018311.686523:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018311.686528:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2574 00000100:00080000:0.0:1734018311.686532:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018311.686536:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018311.686548:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018311.686560:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332955008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018311.686570:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018311.686574:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018311.686575:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018311.686580:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2574 00000100:00080000:0.0:1734018311.686582:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018311.686586:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018311.686592:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018311.686603:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332955136/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018311.686611:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018311.686615:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018311.686616:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018311.686620:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2574 00000100:00080000:0.0:1734018311.686622:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018311.686627:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018311.686632:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018311.686642:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332955264/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018311.686643:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332954880:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018311.686651:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018311.686655:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018311.686656:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018311.686660:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2574 00000100:00080000:0.0:1734018311.686663:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018311.686666:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018311.686671:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018311.686680:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332955392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018311.686720:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018311.686724:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018311.686726:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018311.686730:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2574 00000100:00080000:0.0:1734018311.686749:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018311.686753:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018311.686758:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018311.686763:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018311.686768:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332955520/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018311.686777:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018311.686782:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018311.686783:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018311.686787:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2574 00000100:00100000:1.0:1734018311.686788:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018311/real 1734018311] req@ffff8800a705c000 x1818247332954880/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018311.686791:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018311.686794:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018311.686799:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018311.686808:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332955648/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018311.686816:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332954880/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018311.686827:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018311.686831:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018311.686836:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332954880:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018311.686844:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332955008:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018311.686863:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018311.686870:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018311/real 1734018311] req@ffff8800a705e300 x1818247332955008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018311.686881:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332955008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018311.686889:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018311.686892:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018311.686895:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332955008:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018311.686901:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332955136:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018311.686915:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018311.686921:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018311/real 1734018311] req@ffff8800a705f100 x1818247332955136/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018311.686931:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332955136/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018311.686938:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018311.686941:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018311.686944:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332955136:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018311.686971:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332955264:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018311.686984:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018311.686990:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018311/real 1734018311] req@ffff8800a705dc00 x1818247332955264/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018311.687001:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332955264/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018311.687008:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018311.687011:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018311.687014:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332955264:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018311.687020:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332955392:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018311.687032:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018311.687038:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018311/real 1734018311] req@ffff8800a705d180 x1818247332955392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018311.687048:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332955392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018311.687055:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018311.687058:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018311.687061:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332955392:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018311.687067:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332955520:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018311.687083:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018311.687089:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018311/real 1734018311] req@ffff8800a705c380 x1818247332955520/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018311.687099:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332955520/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018311.687106:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018311.687109:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018311.687112:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332955520:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018311.687117:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332955648:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018311.687130:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018311.687136:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018311/real 1734018311] req@ffff8800a705ed80 x1818247332955648/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018311.687145:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332955648/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018311.687153:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018311.687155:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018311.687158:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332955648:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018312.330476:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018313.332466:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018314.334519:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018315.336492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018316.338432:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018316.694344:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018316.694355:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018316.694359:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018316.694370:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2579 00000100:00080000:0.0:1734018316.694380:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018316.694387:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018316.694423:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332955776/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018316.694456:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018316.694460:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018316.694463:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018316.694467:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2579 00000100:00080000:0.0:1734018316.694470:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018316.694474:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018316.694484:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018316.694495:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332955904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018316.694506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018316.694509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018316.694511:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018316.694515:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2579 00000100:00080000:0.0:1734018316.694517:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018316.694520:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018316.694525:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018316.694534:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332956032/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018316.694542:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018316.694545:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018316.694547:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018316.694550:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2579 00000100:00080000:0.0:1734018316.694552:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018316.694556:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018316.694560:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018316.694565:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332955776:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018316.694568:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332956160/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018316.694577:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018316.694580:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018316.694582:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018316.694586:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2579 00000100:00080000:0.0:1734018316.694589:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018316.694592:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018316.694631:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018316.694642:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332956288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018316.694651:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018316.694654:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018316.694656:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018316.694659:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2579 00000100:00080000:0.0:1734018316.694662:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018316.694666:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018316.694670:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018316.694678:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332956416/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018316.694686:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018316.694690:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018316.694691:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018316.694694:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2579 00000100:00100000:1.0:1734018316.694695:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332955904:192.168.201.152@tcp:38:kworker.0 00000100:00080000:0.0:1734018316.694697:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018316.694700:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018316.694704:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018316.694713:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332956544/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018316.694717:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332956032:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018316.694740:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332956160:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018316.694755:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332956288:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018316.694769:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332956416:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018316.694784:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332956544:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018317.340465:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018318.342489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018319.344501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1734018319.700584:0:1957:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:2.0:1734018319.700647:0:1957:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1734018319.700671:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018319.700725:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018319.700738:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018319.700750:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018319.700761:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018319.700773:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018319.700786:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734018319.700852:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018316/real 1734018319] req@ffff8800a705ea00 x1818247332956160/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018319.700897:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332956160/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018319.700914:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018319.700921:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018319.700928:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332956160:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018319.700941:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018316/real 1734018319] req@ffff8800a705fb80 x1818247332956288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018319.700960:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332956288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018319.700972:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018319.700977:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018319.700983:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332956288:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018319.700994:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018316/real 1734018319] req@ffff8800a705e680 x1818247332956416/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018319.701012:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332956416/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018319.701025:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018319.701030:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018319.701036:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332956416:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018319.701046:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018316/real 1734018319] req@ffff8800a705ca80 x1818247332956544/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018319.701064:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332956544/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018319.701077:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018319.701081:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018319.701086:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332956544:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018319.701097:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018316/real 1734018319] req@ffff8800a705df80 x1818247332955776/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018319.701116:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332955776/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018319.701129:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018319.701133:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018319.701138:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332955776:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018319.701149:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018316/real 1734018319] req@ffff8800a705c700 x1818247332955904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018319.701166:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332955904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018319.701188:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018319.701193:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018319.701198:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332955904:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018319.701208:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018316/real 1734018319] req@ffff8800a705d500 x1818247332956032/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018319.701428:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332956032/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018319.701441:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018319.701445:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018319.701450:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332956032:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018320.346453:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018321.348516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018321.702360:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018321.702370:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018321.702375:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018321.702386:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2584 00000100:00080000:0.0:1734018321.702397:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018321.702405:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018321.702441:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332956672/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018321.702475:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018321.702479:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018321.702482:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018321.702486:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2584 00000100:00080000:0.0:1734018321.702489:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018321.702493:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018321.702504:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018321.702559:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332956800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018321.702570:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018321.702573:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018321.702575:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018321.702579:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2584 00000100:00080000:0.0:1734018321.702582:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018321.702586:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018321.702592:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:1.0:1734018321.702600:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332956672:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018321.702603:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332956928/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018321.702613:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018321.702616:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018321.702618:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018321.702621:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2584 00000100:00080000:0.0:1734018321.702624:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018321.702627:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018321.702632:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018321.702640:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332957056/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018321.702648:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018321.702651:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018321.702652:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018321.702656:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2584 00000100:00080000:0.0:1734018321.702658:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018321.702662:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018321.702666:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1734018321.702668:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018321.702674:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332957184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018321.702682:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018321.702685:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018321.702687:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018321.702689:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018321/real 1734018321] req@ffff8800a705f800 x1818247332956672/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018321.702691:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2584 00000100:00080000:0.0:1734018321.702694:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018321.702697:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018321.702702:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018321.702709:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332957312/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018321.702709:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332956672/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018321.702717:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1734018321.702719:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018321.702720:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018321.702722:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734018321.702722:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:0.0:1734018321.702725:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2584 00000100:00080000:0.0:1734018321.702727:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018321.702727:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332956672:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018321.702731:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018321.702735:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018321.702735:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332956800:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018321.702743:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332957440/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018321.702752:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018321.702760:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018321/real 1734018321] req@ffff8800a705d880 x1818247332956800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018321.702771:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332956800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018321.702779:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018321.702781:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018321.702784:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332956800:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018321.702808:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332956928:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018321.702822:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018321.702828:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018321/real 1734018321] req@ffff8800a705d500 x1818247332956928/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018321.702838:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332956928/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018321.702846:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018321.702848:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018321.702852:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332956928:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018321.702858:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332957056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018321.702869:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018321.702875:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018321/real 1734018321] req@ffff8800a705c700 x1818247332957056/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018321.702885:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332957056/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018321.702899:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018321.702902:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018321.702905:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332957056:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018321.702910:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332957184:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018321.702922:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018321.702928:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018321/real 1734018321] req@ffff8800a705df80 x1818247332957184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018321.702938:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332957184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018321.702945:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018321.702947:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018321.702950:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332957184:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018321.702956:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332957312:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018321.702969:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018321.702974:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018321/real 1734018321] req@ffff8800a705ca80 x1818247332957312/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018321.702984:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332957312/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018321.702991:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018321.702993:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018321.702996:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332957312:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018321.703003:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332957440:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018321.703015:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018321.703021:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018321/real 1734018321] req@ffff8800a705e680 x1818247332957440/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018321.703030:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332957440/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018321.703038:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018321.703040:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018321.703043:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332957440:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018322.350490:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018323.352514:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018324.354510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018325.356509:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018326.358484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018326.710367:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018326.710376:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018326.710381:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018326.710392:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2589 00000100:00080000:0.0:1734018326.710402:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018326.710410:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018326.710484:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332957568/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018326.710525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018326.710530:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018326.710532:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018326.710536:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2589 00000100:00080000:0.0:1734018326.710540:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018326.710544:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018326.710555:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018326.710566:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332957696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018326.710576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018326.710580:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018326.710581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018326.710585:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2589 00000100:00080000:0.0:1734018326.710587:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018326.710591:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018326.710596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018326.710603:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332957824/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018326.710612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018326.710615:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018326.710616:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018326.710620:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2589 00000100:00080000:0.0:1734018326.710622:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018326.710626:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018326.710630:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018326.710639:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332957952/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018326.710641:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332957568:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018326.710647:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018326.710650:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018326.710652:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018326.710655:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2589 00000100:00080000:0.0:1734018326.710658:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018326.710661:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018326.710665:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018326.710673:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332958080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018326.710681:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018326.710684:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018326.710685:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018326.710689:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2589 00000100:00080000:0.0:1734018326.710693:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018326.710697:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018326.710701:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018326.710708:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332958208/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018326.710708:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018326.710716:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018326.710720:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018326.710721:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018326.710725:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2589 00000100:00080000:0.0:1734018326.710728:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018326.710730:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018326/real 1734018326] req@ffff8800a705fb80 x1818247332957568/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018326.710731:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018326.710735:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018326.710745:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332958336/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018326.710749:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332957568/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018326.710760:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018326.710763:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018326.710768:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332957568:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018326.710775:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332957696:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018326.710792:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018326.710798:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018326/real 1734018326] req@ffff8800a705ea00 x1818247332957696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018326.710818:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332957696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018326.710825:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018326.710828:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018326.710831:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332957696:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018326.710837:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332957824:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018326.710851:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018326.710857:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018326/real 1734018326] req@ffff8800a705ed80 x1818247332957824/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018326.710867:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332957824/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018326.710874:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018326.710877:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018326.710880:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332957824:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018326.710904:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332957952:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018326.710916:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018326.710922:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018326/real 1734018326] req@ffff8800a705c380 x1818247332957952/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018326.710933:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332957952/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018326.710940:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018326.710943:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018326.710946:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332957952:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018326.710952:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332958080:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018326.710964:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018326.710970:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018326/real 1734018326] req@ffff8800a705d180 x1818247332958080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018326.710980:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332958080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018326.710988:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018326.710991:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018326.710993:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332958080:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018326.710999:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332958208:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018326.711010:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018326.711016:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018326/real 1734018326] req@ffff8800a705dc00 x1818247332958208/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018326.711026:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332958208/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018326.711033:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018326.711035:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018326.711038:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332958208:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018326.711043:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332958336:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018326.711059:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018326.711065:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018326/real 1734018326] req@ffff8800a705f100 x1818247332958336/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018326.711075:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332958336/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018326.711082:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018326.711084:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018326.711087:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332958336:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018327.360511:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018328.362491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018329.364489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018330.366510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018331.368518:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018331.718414:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018331.718423:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018331.718428:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018331.718439:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2594 00000100:00080000:0.0:1734018331.718448:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018331.718456:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018331.718494:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332958464/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018331.718527:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018331.718531:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018331.718533:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018331.718538:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2594 00000100:00080000:0.0:1734018331.718541:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018331.718545:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018331.718555:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018331.718565:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332958592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018331.718576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018331.718579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018331.718581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018331.718584:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2594 00000100:00080000:0.0:1734018331.718587:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018331.718590:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018331.718595:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018331.718605:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332958720/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018331.718614:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018331.718617:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018331.718618:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018331.718622:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2594 00000100:00080000:0.0:1734018331.718624:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018331.718627:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018331.718632:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018331.718640:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332958848/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018331.718643:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332958464:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018331.718649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018331.718652:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018331.718654:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018331.718657:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2594 00000100:00080000:0.0:1734018331.718660:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018331.718663:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018331.718668:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018331.718676:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332958976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018331.718684:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018331.718687:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018331.718688:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018331.718692:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2594 00000100:00080000:0.0:1734018331.718695:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018331.718699:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018331.718703:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018331.718710:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018331.718711:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332959104/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018331.718719:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018331.718722:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018331.718723:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018331.718726:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2594 00000100:00080000:0.0:1734018331.718729:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018331.718733:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018331.718733:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018331/real 1734018331] req@ffff8800a705e300 x1818247332958464/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018331.718737:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018331.718745:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332959232/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018331.718752:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332958464/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018331.718764:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018331.718767:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018331.718771:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332958464:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018331.718779:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332958592:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018331.718796:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018331.718802:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018331/real 1734018331] req@ffff8800a705c000 x1818247332958592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018331.718813:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332958592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018331.718822:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018331.718824:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018331.718828:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332958592:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018331.718833:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332958720:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018331.718847:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018331.718853:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018331/real 1734018331] req@ffff8800a705f100 x1818247332958720/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018331.718863:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332958720/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018331.718870:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018331.718873:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018331.718876:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332958720:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018331.718901:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332958848:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018331.718913:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018331.718920:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018331/real 1734018331] req@ffff8800a705dc00 x1818247332958848/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018331.718930:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332958848/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018331.718937:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018331.718939:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018331.718942:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332958848:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018331.718949:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332958976:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018331.718961:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018331.718967:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018331/real 1734018331] req@ffff8800a705d180 x1818247332958976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018331.718978:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332958976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018331.718986:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018331.718988:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018331.718991:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332958976:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018331.718996:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332959104:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018331.719008:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018331.719013:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018331/real 1734018331] req@ffff8800a705c380 x1818247332959104/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018331.719023:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332959104/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018331.719030:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018331.719032:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018331.719035:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332959104:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018331.719040:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332959232:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018331.719053:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018331.719059:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018331/real 1734018331] req@ffff8800a705ed80 x1818247332959232/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018331.719069:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332959232/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018331.719077:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018331.719079:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018331.719082:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332959232:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018332.370506:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018333.372490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018334.374523:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018335.376510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018336.378470:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018336.726365:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018336.726374:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018336.726378:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018336.726389:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2599 00000100:00080000:0.0:1734018336.726398:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018336.726405:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018336.726442:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332959360/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018336.726474:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018336.726478:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018336.726480:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018336.726485:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2599 00000100:00080000:0.0:1734018336.726488:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018336.726492:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018336.726503:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018336.726514:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332959488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018336.726524:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018336.726527:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018336.726529:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018336.726532:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2599 00000100:00080000:0.0:1734018336.726535:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018336.726539:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018336.726544:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018336.726552:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332959616/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018336.726561:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018336.726564:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018336.726565:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018336.726569:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2599 00000100:00080000:0.0:1734018336.726571:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018336.726574:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018336.726579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018336.726584:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332959360:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018336.726586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332959744/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018336.726595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018336.726598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018336.726600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018336.726603:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2599 00000100:00080000:0.0:1734018336.726606:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018336.726609:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018336.726613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018336.726620:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332959872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018336.726629:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018336.726632:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018336.726634:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018336.726638:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2599 00000100:00080000:0.0:1734018336.726641:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018336.726644:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018336.726649:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018336.726652:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018336.726656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332960000/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018336.726664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018336.726667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018336.726669:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018336.726672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2599 00000100:00080000:0.0:1734018336.726674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018336.726675:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018336/real 1734018336] req@ffff8800a705ea00 x1818247332959360/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018336.726678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018336.726682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018336.726690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332960128/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018336.726694:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332959360/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018336.726705:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018336.726708:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018336.726713:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332959360:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018336.726720:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332959488:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018336.726737:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018336.726744:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018336/real 1734018336] req@ffff8800a705fb80 x1818247332959488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018336.726754:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332959488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018336.726762:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018336.726766:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018336.726769:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332959488:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018336.726776:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332959616:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018336.726789:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018336.726795:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018336/real 1734018336] req@ffff8800a705e680 x1818247332959616/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018336.726805:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332959616/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018336.726812:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018336.726815:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018336.726818:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332959616:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018336.726842:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332959744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018336.726854:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018336.726860:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018336/real 1734018336] req@ffff8800a705ca80 x1818247332959744/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018336.726870:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332959744/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018336.726878:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018336.726880:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018336.726883:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332959744:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018336.726889:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332959872:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018336.726904:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018336.726910:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018336/real 1734018336] req@ffff8800a705df80 x1818247332959872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018336.726919:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332959872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018336.726928:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018336.726930:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018336.726933:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332959872:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018336.726939:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332960000:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018336.726950:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018336.726956:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018336/real 1734018336] req@ffff8800a705c700 x1818247332960000/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018336.726966:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332960000/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018336.726973:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018336.726975:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018336.726978:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332960000:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018336.726984:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332960128:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018336.726996:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018336.727001:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018336/real 1734018336] req@ffff8800a705d500 x1818247332960128/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018336.727018:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332960128/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018336.727025:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018336.727027:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018336.727030:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332960128:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018337.380475:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018338.382450:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018339.384522:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018340.386513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018341.388502:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018341.734352:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018341.734362:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018341.734367:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018341.734378:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2604 00000100:00080000:0.0:1734018341.734388:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018341.734396:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018341.734431:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332960256/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018341.734473:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018341.734477:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018341.734479:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018341.734483:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2604 00000100:00080000:0.0:1734018341.734487:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018341.734491:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018341.734500:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018341.734511:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332960384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018341.734521:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018341.734524:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018341.734526:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018341.734530:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2604 00000100:00080000:0.0:1734018341.734532:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018341.734536:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018341.734540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018341.734550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332960512/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018341.734559:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018341.734562:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018341.734563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018341.734567:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2604 00000100:00080000:0.0:1734018341.734569:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018341.734573:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018341.734577:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018341.734580:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332960256:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018341.734585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332960640/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018341.734594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018341.734598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018341.734599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018341.734603:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2604 00000100:00080000:0.0:1734018341.734605:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018341.734608:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018341.734612:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018341.734620:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332960768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018341.734628:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018341.734631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018341.734632:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018341.734636:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2604 00000100:00080000:0.0:1734018341.734639:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018341.734642:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018341.734646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018341.734646:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018341.734653:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332960896/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018341.734662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018341.734665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018341.734667:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018341.734668:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018341/real 1734018341] req@ffff8800a705d880 x1818247332960256/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018341.734671:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2604 00000100:00080000:0.0:1734018341.734674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018341.734677:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018341.734681:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018341.734687:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332960256/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018341.734689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332961024/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018341.734698:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018341.734701:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018341.734706:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332960256:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018341.734713:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332960384:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018341.734730:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018341.734736:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018341/real 1734018341] req@ffff8800a705f800 x1818247332960384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018341.734748:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332960384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018341.734755:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018341.734759:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018341.734762:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332960384:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018341.734768:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332960512:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018341.734781:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018341.734787:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018341/real 1734018341] req@ffff8800a705d500 x1818247332960512/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018341.734797:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332960512/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018341.734805:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018341.734808:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018341.734811:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332960512:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018341.734836:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332960640:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018341.734848:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018341.734854:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018341/real 1734018341] req@ffff8800a705c700 x1818247332960640/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018341.734864:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332960640/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018341.734872:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018341.734874:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018341.734877:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332960640:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018341.734882:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332960768:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018341.734898:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018341.734904:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018341/real 1734018341] req@ffff8800a705df80 x1818247332960768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018341.734914:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332960768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018341.734921:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018341.734923:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018341.734926:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332960768:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018341.734932:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332960896:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018341.734944:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018341.734949:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018341/real 1734018341] req@ffff8800a705ca80 x1818247332960896/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018341.734959:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332960896/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018341.734967:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018341.734969:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018341.734972:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332960896:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018341.734978:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332961024:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018341.734989:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018341.734995:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018341/real 1734018341] req@ffff8800a705e680 x1818247332961024/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018341.735011:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332961024/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018341.735019:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018341.735021:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018341.735024:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332961024:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018342.390462:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018343.392490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018344.394517:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018345.396493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018346.398494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018346.742367:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018346.742379:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018346.742385:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018346.742399:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2609 00000100:00080000:0.0:1734018346.742411:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018346.742420:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018346.742470:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332961152/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018346.742508:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018346.742512:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018346.742514:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018346.742520:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2609 00000100:00080000:0.0:1734018346.742523:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018346.742527:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018346.742539:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018346.742551:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332961280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018346.742563:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018346.742566:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018346.742568:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018346.742572:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2609 00000100:00080000:0.0:1734018346.742574:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018346.742578:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018346.742583:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018346.742592:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332961408/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018346.742601:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018346.742604:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018346.742605:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018346.742609:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2609 00000100:00080000:0.0:1734018346.742611:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018346.742615:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018346.742619:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018346.742628:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332961536/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018346.742638:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018346.742639:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332961152:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018346.742641:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018346.742643:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018346.742646:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2609 00000100:00080000:0.0:1734018346.742650:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018346.742654:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018346.742658:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018346.742667:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332961664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018346.742676:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018346.742679:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018346.742680:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018346.742684:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2609 00000100:00080000:0.0:1734018346.742687:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018346.742691:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018346.742695:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018346.742703:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332961792/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018346.742711:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018346.742714:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018346.742715:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734018346.742718:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018346.742719:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2609 00000100:00080000:0.0:1734018346.742722:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018346.742726:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018346.742730:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018346.742740:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332961920/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018346.742744:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018346/real 1734018346] req@ffff8800a705fb80 x1818247332961152/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018346.742764:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332961152/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018346.742776:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018346.742780:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018346.742785:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332961152:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018346.742793:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332961280:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018346.742812:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018346.742819:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018346/real 1734018346] req@ffff8800a705ea00 x1818247332961280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018346.742830:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332961280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018346.742838:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018346.742843:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018346.742846:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332961280:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018346.742853:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332961408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018346.742867:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018346.742874:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018346/real 1734018346] req@ffff8800a705ed80 x1818247332961408/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018346.742884:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332961408/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018346.742892:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018346.742894:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018346.742898:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332961408:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018346.742925:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332961536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018346.742937:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018346.742943:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018346/real 1734018346] req@ffff8800a705c380 x1818247332961536/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018346.742954:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332961536/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018346.742962:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018346.742964:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018346.742967:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332961536:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018346.742973:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332961664:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018346.742989:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018346.742995:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018346/real 1734018346] req@ffff8800a705d180 x1818247332961664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018346.743005:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332961664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018346.743012:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018346.743015:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018346.743018:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332961664:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018346.743023:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332961792:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018346.743036:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018346.743041:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018346/real 1734018346] req@ffff8800a705dc00 x1818247332961792/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018346.743051:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332961792/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018346.743059:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018346.743061:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018346.743064:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332961792:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018346.743070:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332961920:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018346.743082:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018346.743136:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018346/real 1734018346] req@ffff8800a705f100 x1818247332961920/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018346.743154:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332961920/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018346.743162:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018346.743164:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018346.743167:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332961920:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018347.400451:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018348.402460:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018349.404518:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018350.406448:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018351.408481:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018351.750348:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018351.750359:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018351.750363:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018351.750374:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2614 00000100:00080000:0.0:1734018351.750384:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018351.750392:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018351.750428:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332962048/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018351.750461:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018351.750465:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018351.750467:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018351.750472:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2614 00000100:00080000:0.0:1734018351.750475:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018351.750479:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018351.750489:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018351.750499:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332962176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018351.750509:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018351.750513:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018351.750514:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018351.750518:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2614 00000100:00080000:0.0:1734018351.750521:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018351.750524:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018351.750530:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018351.750540:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332962304/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018351.750549:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018351.750552:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018351.750553:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018351.750557:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2614 00000100:00080000:0.0:1734018351.750559:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018351.750563:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018351.750567:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018351.750568:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332962048:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018351.750575:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332962432/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018351.750584:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018351.750587:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018351.750588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018351.750592:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2614 00000100:00080000:0.0:1734018351.750594:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018351.750598:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018351.750602:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018351.750610:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332962560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018351.750618:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018351.750621:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018351.750623:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018351.750626:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2614 00000100:00080000:0.0:1734018351.750629:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018351.750633:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018351.750636:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734018351.750637:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018351.750644:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332962688/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018351.750652:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018351.750655:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018351.750657:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018351.750658:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018351/real 1734018351] req@ffff8800a705c000 x1818247332962048/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018351.750660:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2614 00000100:00080000:0.0:1734018351.750662:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018351.750665:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018351.750670:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018351.750677:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332962816/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018351.750678:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332962048/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018351.750690:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018351.750693:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018351.750697:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332962048:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018351.750705:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332962176:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018351.750721:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018351.750727:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018351/real 1734018351] req@ffff8800a705e300 x1818247332962176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018351.750738:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332962176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018351.750746:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018351.750749:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018351.750753:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332962176:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018351.750759:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332962304:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018351.750773:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018351.750778:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018351/real 1734018351] req@ffff8800a705f100 x1818247332962304/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018351.750788:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332962304/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018351.750796:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018351.750799:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018351.750802:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332962304:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018351.750826:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332962432:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018351.750839:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018351.750845:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018351/real 1734018351] req@ffff8800a705dc00 x1818247332962432/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018351.750855:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332962432/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018351.750863:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018351.750865:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018351.750868:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332962432:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018351.750873:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332962560:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018351.750888:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018351.750894:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018351/real 1734018351] req@ffff8800a705d180 x1818247332962560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018351.750905:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332962560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018351.750912:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018351.750914:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018351.750917:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332962560:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018351.750923:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332962688:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018351.750934:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018351.750939:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018351/real 1734018351] req@ffff8800a705c380 x1818247332962688/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018351.750949:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332962688/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018351.750956:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018351.750958:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018351.750961:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332962688:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018351.750967:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332962816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018351.750979:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018351.750984:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018351/real 1734018351] req@ffff8800a705ed80 x1818247332962816/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018351.751039:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332962816/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018351.751048:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018351.751050:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018351.751054:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332962816:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018352.410482:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018353.412467:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018354.414488:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018355.416493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018356.418454:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018356.758340:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018356.758350:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018356.758355:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018356.758366:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2619 00000100:00080000:0.0:1734018356.758375:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018356.758383:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018356.758417:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332962944/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018356.758457:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018356.758462:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018356.758463:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018356.758468:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2619 00000100:00080000:0.0:1734018356.758471:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018356.758475:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018356.758484:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018356.758495:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332963072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018356.758504:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018356.758508:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018356.758509:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018356.758513:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2619 00000100:00080000:0.0:1734018356.758516:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018356.758519:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018356.758524:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018356.758532:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332963200/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018356.758540:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018356.758543:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018356.758545:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018356.758548:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2619 00000100:00080000:0.0:1734018356.758550:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018356.758554:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018356.758559:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018356.758566:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332962944:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018356.758567:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332963328/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018356.758576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018356.758579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018356.758581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018356.758584:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2619 00000100:00080000:0.0:1734018356.758587:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018356.758590:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018356.758594:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018356.758602:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332963456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018356.758610:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018356.758613:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018356.758614:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018356.758618:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2619 00000100:00080000:0.0:1734018356.758621:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018356.758624:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018356.758628:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018356.758635:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018356.758636:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332963584/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018356.758644:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018356.758648:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018356.758650:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018356.758653:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2619 00000100:00080000:0.0:1734018356.758656:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018356.758657:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018356/real 1734018356] req@ffff8800a705ea00 x1818247332962944/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018356.758659:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018356.758663:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018356.758673:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332963712/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018356.758676:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332962944/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018356.758687:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018356.758690:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018356.758695:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332962944:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018356.758702:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332963072:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018356.758719:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018356.758725:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018356/real 1734018356] req@ffff8800a705fb80 x1818247332963072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018356.758736:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332963072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018356.758744:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018356.758748:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018356.758751:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332963072:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018356.758757:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332963200:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018356.758771:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018356.758777:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018356/real 1734018356] req@ffff8800a705e680 x1818247332963200/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018356.758787:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332963200/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018356.758795:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018356.758797:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018356.758800:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332963200:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018356.758825:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332963328:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018356.758837:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018356.758843:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018356/real 1734018356] req@ffff8800a705ca80 x1818247332963328/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018356.758853:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332963328/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018356.758861:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018356.758863:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018356.758866:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332963328:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018356.758872:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332963456:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018356.758887:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018356.758892:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018356/real 1734018356] req@ffff8800a705df80 x1818247332963456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018356.758903:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332963456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018356.758910:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018356.758912:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018356.758953:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332963456:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018356.758960:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332963584:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018356.758976:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018356.758983:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018356/real 1734018356] req@ffff8800a705c700 x1818247332963584/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018356.758993:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332963584/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018356.759001:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018356.759003:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018356.759006:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332963584:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018356.759011:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332963712:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018356.759023:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018356.759029:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018356/real 1734018356] req@ffff8800a705d500 x1818247332963712/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018356.759045:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332963712/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018356.759053:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018356.759055:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018356.759058:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332963712:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018357.420503:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018358.422471:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018359.424486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018360.426511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018361.428502:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018361.766401:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018361.766413:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018361.766417:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018361.766429:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2624 00000100:00080000:0.0:1734018361.766440:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018361.766449:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018361.766487:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332963840/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018361.766525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018361.766530:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018361.766532:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018361.766536:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2624 00000100:00080000:0.0:1734018361.766539:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018361.766543:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018361.766554:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018361.766564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332963968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018361.766589:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018361.766591:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018361.766592:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018361.766594:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2624 00000100:00080000:0.0:1734018361.766596:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018361.766598:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018361.766601:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018361.766608:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332964096/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018361.766613:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018361.766615:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018361.766616:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018361.766618:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2624 00000100:00080000:0.0:1734018361.766619:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018361.766621:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018361.766624:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018361.766629:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332964224/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018361.766634:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018361.766636:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018361.766637:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018361.766639:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2624 00000100:00080000:0.0:1734018361.766641:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018361.766643:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018361.766643:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332963840:192.168.201.152@tcp:250:kworker.0 00000008:00100000:0.0:1734018361.766646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018361.766651:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332964352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018361.766656:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018361.766657:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018361.766658:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018361.766660:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2624 00000100:00080000:0.0:1734018361.766662:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018361.766664:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018361.766666:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018361.766670:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332964480/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018361.766675:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018361.766677:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018361.766678:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018361.766680:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2624 00000100:00080000:0.0:1734018361.766681:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018361.766683:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018361.766685:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018361.766690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332964608/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018361.766691:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018361.766706:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018361/real 1734018361] req@ffff8800a705f800 x1818247332963840/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018361.766718:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332963840/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018361.766725:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018361.766727:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018361.766729:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332963840:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018361.766734:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332963968:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018361.766744:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018361.766748:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018361/real 1734018361] req@ffff8800a705d880 x1818247332963968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018361.766755:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332963968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018361.766759:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018361.766761:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018361.766763:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332963968:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018361.766767:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332964096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018361.766775:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018361.766778:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018361/real 1734018361] req@ffff8800a705d500 x1818247332964096/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018361.766784:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332964096/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018361.766789:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018361.766791:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018361.766793:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332964096:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018361.766809:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332964224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018361.766817:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018361.766820:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018361/real 1734018361] req@ffff8800a705c700 x1818247332964224/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018361.766826:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332964224/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018361.766861:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018361.766862:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018361.766864:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332964224:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018361.766868:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332964352:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018361.766880:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018361.766885:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018361/real 1734018361] req@ffff8800a705df80 x1818247332964352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018361.766891:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332964352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018361.766896:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018361.766897:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018361.766899:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332964352:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018361.766902:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332964480:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018361.766909:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018361.766912:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018361/real 1734018361] req@ffff8800a705ca80 x1818247332964480/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018361.766918:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332964480/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018361.766922:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018361.766923:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018361.766925:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332964480:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018361.766928:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332964608:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018361.766935:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018361.766939:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018361/real 1734018361] req@ffff8800a705e680 x1818247332964608/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018361.766948:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332964608/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018361.766953:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018361.766954:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018361.766956:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332964608:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018362.430482:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018363.432486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018364.434494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018365.436462:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018366.438481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018366.774364:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018366.774373:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018366.774378:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018366.774390:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2629 00000100:00080000:0.0:1734018366.774399:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018366.774406:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018366.774442:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332964736/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018366.774477:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018366.774482:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018366.774484:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018366.774488:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2629 00000100:00080000:0.0:1734018366.774492:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018366.774496:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018366.774505:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018366.774516:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332964864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018366.774526:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018366.774529:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018366.774531:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018366.774535:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2629 00000100:00080000:0.0:1734018366.774538:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018366.774541:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018366.774546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018366.774554:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332964992/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018366.774563:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018366.774566:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018366.774568:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018366.774571:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2629 00000100:00080000:0.0:1734018366.774573:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018366.774577:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018366.774581:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018366.774589:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332965120/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018366.774597:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018366.774597:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332964736:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018366.774601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018366.774602:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018366.774606:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2629 00000100:00080000:0.0:1734018366.774608:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018366.774611:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018366.774615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018366.774623:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332965248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018366.774631:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018366.774634:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018366.774636:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018366.774640:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2629 00000100:00080000:0.0:1734018366.774652:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018366.774656:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018366.774661:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018366.774665:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018366.774669:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332965376/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018366.774677:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018366.774680:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018366.774682:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018366.774686:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2629 00000100:00100000:1.0:1734018366.774687:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018366/real 1734018366] req@ffff8800a705fb80 x1818247332964736/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018366.774688:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018366.774692:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018366.774696:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018366.774705:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332965504/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018366.774707:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332964736/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018366.774717:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018366.774720:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018366.774725:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332964736:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018366.774732:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332964864:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018366.774796:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018366.774804:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018366/real 1734018366] req@ffff8800a705ea00 x1818247332964864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018366.774816:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332964864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018366.774824:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018366.774828:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018366.774831:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332964864:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018366.774838:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332964992:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018366.774853:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018366.774859:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018366/real 1734018366] req@ffff8800a705ed80 x1818247332964992/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018366.774868:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332964992/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018366.774876:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018366.774879:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018366.774882:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332964992:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018366.774907:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332965120:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018366.774920:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018366.774925:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018366/real 1734018366] req@ffff8800a705c380 x1818247332965120/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018366.774936:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332965120/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018366.774944:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018366.774946:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018366.774949:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332965120:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018366.774954:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332965248:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018366.774969:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018366.774975:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018366/real 1734018366] req@ffff8800a705d180 x1818247332965248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018366.774986:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332965248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018366.774993:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018366.774996:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018366.774999:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332965248:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018366.775004:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332965376:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018366.775016:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018366.775022:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018366/real 1734018366] req@ffff8800a705dc00 x1818247332965376/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018366.775032:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332965376/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018366.775039:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018366.775042:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018366.775045:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332965376:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018366.775049:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332965504:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018366.775061:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018366.775067:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018366/real 1734018366] req@ffff8800a705f100 x1818247332965504/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018366.775083:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332965504/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018366.775091:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018366.775093:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018366.775096:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332965504:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018367.440489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018368.442486:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018369.444457:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018370.446452:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018371.448480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018371.782363:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018371.782373:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018371.782377:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018371.782389:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2634 00000100:00080000:0.0:1734018371.782399:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018371.782406:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018371.782442:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332965632/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018371.782487:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018371.782493:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018371.782496:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018371.782502:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2634 00000100:00080000:0.0:1734018371.782507:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018371.782514:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018371.782527:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018371.782545:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332965760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018371.782558:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332965632:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018371.782560:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018371.782565:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018371.782568:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018371.782573:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2634 00000100:00080000:0.0:1734018371.782578:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018371.782583:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018371.782590:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018371.782604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332965888/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018371.782616:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018371.782621:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018371.782623:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018371.782628:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2634 00000800:00000100:1.0:1734018371.782630:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018371.782632:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018371.782637:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018371.782644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018371.782653:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018371/real 1734018371] req@ffff8800a705e300 x1818247332965632/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018371.782656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332966016/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018371.782715:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332965632/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018371.782720:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018371.782725:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018371.782728:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734018371.782730:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018371.782734:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2634 00000100:00080000:1.0:1734018371.782734:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:0.0:1734018371.782738:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018371.782740:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332965632:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018371.782744:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018371.782751:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018371.782766:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332966144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018371.782768:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332965760:192.168.201.152@tcp:38:kworker.0 00000100:00080000:0.0:1734018371.782780:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018371.782784:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018371.782787:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018371.782792:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2634 00000100:00080000:0.0:1734018371.782796:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018371.782797:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018371.782801:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018371.782809:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018371.782810:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018371/real 1734018371] req@ffff8800a705c000 x1818247332965760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018371.782821:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332966272/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018371.782828:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332965760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018371.782835:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1734018371.782840:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018371.782841:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018371.782843:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734018371.782845:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00080000:0.0:1734018371.782848:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2634 00000100:00100000:1.0:1734018371.782850:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332965760:192.168.201.152@tcp:38:kworker.0 00000100:00080000:0.0:1734018371.782853:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018371.782858:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018371.782859:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332965888:192.168.201.152@tcp:8:kworker.0 00000008:00100000:0.0:1734018371.782864:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018371.782875:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332966400/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018371.782882:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018371.782893:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018371/real 1734018371] req@ffff8800a705f100 x1818247332965888/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018371.782911:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332965888/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018371.782923:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018371.782926:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018371.782932:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332965888:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018371.782940:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332966016:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018371.782966:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018371.782978:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018371/real 1734018371] req@ffff8800a705dc00 x1818247332966016/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018371.782989:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332966016/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018371.782997:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018371.782999:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018371.783002:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332966016:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018371.783019:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332966144:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018371.783038:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018371.783045:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018371/real 1734018371] req@ffff8800a705d180 x1818247332966144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018371.783055:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332966144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018371.783063:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018371.783065:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018371.783068:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332966144:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018371.783073:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332966272:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018371.783084:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018371.783090:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018371/real 1734018371] req@ffff8800a705c380 x1818247332966272/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018371.783100:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332966272/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018371.783107:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018371.783109:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018371.783112:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332966272:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018371.783117:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332966400:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018371.783129:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018371.783135:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018371/real 1734018371] req@ffff8800a705ed80 x1818247332966400/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018371.783151:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332966400/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018371.783159:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018371.783161:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018371.783164:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332966400:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018372.450447:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018373.452501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018374.454465:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018375.456499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018376.458513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018376.790363:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018376.790373:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018376.790377:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018376.790388:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2639 00000100:00080000:0.0:1734018376.790398:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018376.790405:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018376.790441:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332966528/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018376.790476:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018376.790480:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018376.790482:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018376.790487:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2639 00000100:00080000:0.0:1734018376.790490:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018376.790493:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018376.790503:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018376.790514:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332966656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018376.790525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018376.790529:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018376.790530:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018376.790534:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2639 00000100:00080000:0.0:1734018376.790536:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018376.790539:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018376.790544:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018376.790552:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332966784/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018376.790561:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018376.790564:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018376.790565:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018376.790568:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2639 00000100:00080000:0.0:1734018376.790571:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018376.790603:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018376.790608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018376.790617:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332966912/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018376.790626:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018376.790626:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332966528:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018376.790629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018376.790631:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018376.790635:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2639 00000100:00080000:0.0:1734018376.790638:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018376.790641:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018376.790646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018376.790653:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332967040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018376.790662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018376.790665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018376.790666:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018376.790670:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2639 00000100:00080000:0.0:1734018376.790682:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018376.790686:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018376.790690:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018376.790693:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018376.790698:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332967168/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018376.790707:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018376.790710:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018376.790711:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018376.790715:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2639 00000100:00100000:1.0:1734018376.790715:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018376/real 1734018376] req@ffff8800a705ea00 x1818247332966528/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018376.790718:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018376.790721:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018376.790725:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018376.790732:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332967296/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018376.790734:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332966528/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018376.790745:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018376.790749:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018376.790753:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332966528:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018376.790760:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332966656:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018376.790777:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018376.790783:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018376/real 1734018376] req@ffff8800a705fb80 x1818247332966656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018376.790795:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332966656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018376.790802:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018376.790806:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018376.790809:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332966656:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018376.790816:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332966784:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018376.790830:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018376.790835:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018376/real 1734018376] req@ffff8800a705e680 x1818247332966784/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018376.790845:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332966784/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018376.790853:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018376.790856:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018376.790859:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332966784:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018376.790884:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332966912:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018376.790896:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018376.790902:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018376/real 1734018376] req@ffff8800a705ca80 x1818247332966912/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018376.790913:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332966912/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018376.790922:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018376.790924:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018376.790927:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332966912:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018376.790932:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332967040:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018376.790948:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018376.790954:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018376/real 1734018376] req@ffff8800a705df80 x1818247332967040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018376.790965:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332967040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018376.790972:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018376.790974:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018376.790977:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332967040:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018376.790982:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332967168:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018376.790994:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018376.790999:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018376/real 1734018376] req@ffff8800a705c700 x1818247332967168/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018376.791010:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332967168/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018376.791018:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018376.791020:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018376.791023:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332967168:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018376.791028:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332967296:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018376.791040:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018376.791045:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018376/real 1734018376] req@ffff8800a705d500 x1818247332967296/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018376.791061:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332967296/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018376.791069:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018376.791071:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018376.791074:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332967296:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018377.460500:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018378.462379:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018379.464432:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018380.466404:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018381.468405:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018381.798346:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018381.798360:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018381.798365:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018381.798380:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2644 00000100:00080000:0.0:1734018381.798392:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018381.798403:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018381.798451:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332967424/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018381.798519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018381.798524:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018381.798526:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018381.798532:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2644 00000100:00080000:0.0:1734018381.798536:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018381.798540:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018381.798553:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018381.798566:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332967552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018381.798577:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018381.798580:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018381.798582:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018381.798586:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2644 00000100:00080000:0.0:1734018381.798588:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018381.798592:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018381.798598:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018381.798609:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332967680/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018381.798619:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018381.798622:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018381.798624:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018381.798624:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332967424:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018381.798628:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2644 00000100:00080000:0.0:1734018381.798631:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018381.798635:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018381.798640:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018381.798649:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332967808/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018381.798657:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018381.798660:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018381.798662:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018381.798665:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2644 00000100:00080000:0.0:1734018381.798668:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018381.798671:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018381.798675:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018381.798684:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332967936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018381.798692:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018381.798695:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018381.798697:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018381.798701:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2644 00000100:00080000:0.0:1734018381.798705:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018381.798708:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018381.798713:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018381.798721:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332968064/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018381.798729:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018381.798731:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332967552:192.168.201.152@tcp:38:kworker.0 00000100:00080000:0.0:1734018381.798732:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018381.798733:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018381.798737:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2644 00000100:00080000:0.0:1734018381.798740:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018381.798743:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018381.798748:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018381.798756:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332968192/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018381.798763:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332967680:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018381.798780:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332967808:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018381.798794:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332967936:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018381.798810:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332968064:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018381.798827:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332968192:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018382.470412:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018383.472399:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018384.474413:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734018384.804632:0:1958:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:3.0:1734018384.804691:0:1958:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734018384.804714:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734018384.804758:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734018384.804769:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734018384.804779:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734018384.804788:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734018384.804798:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734018384.804806:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734018384.804874:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018381/real 1734018384] req@ffff8800a705e680 x1818247332968192/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018384.804908:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332968192/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018384.804936:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018384.804942:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018384.804949:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332968192:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018384.804962:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018381/real 1734018384] req@ffff8800a705d500 x1818247332967680/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018384.804981:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332967680/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018384.804994:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018384.804998:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018384.805004:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332967680:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018384.805014:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018381/real 1734018384] req@ffff8800a705c700 x1818247332967808/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018384.805031:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332967808/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018384.805043:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018384.805047:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018384.805051:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332967808:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018384.805061:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018381/real 1734018384] req@ffff8800a705df80 x1818247332967936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018384.805078:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332967936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018384.805091:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018384.805094:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018384.805100:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332967936:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018384.805115:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018381/real 1734018384] req@ffff8800a705ca80 x1818247332968064/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018384.805133:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332968064/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018384.805146:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018384.805149:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018384.805155:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332968064:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018384.805164:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018381/real 1734018384] req@ffff8800a705d880 x1818247332967424/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018384.805183:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332967424/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018384.805195:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018384.805200:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018384.805205:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332967424:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018384.805367:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018381/real 1734018384] req@ffff8800a705f800 x1818247332967552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018384.805390:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332967552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018384.805404:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018384.805408:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018384.805414:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332967552:192.168.201.152@tcp:38:kworker.0 00010000:00010000:3.0:1734018385.476504:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018386.478493:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018386.806335:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018386.806344:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018386.806349:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018386.806360:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2649 00000100:00080000:0.0:1734018386.806369:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018386.806377:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018386.806449:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332968320/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018386.806488:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018386.806492:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018386.806494:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018386.806498:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2649 00000100:00080000:0.0:1734018386.806502:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018386.806506:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018386.806517:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018386.806528:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332968448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018386.806538:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018386.806541:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018386.806543:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018386.806546:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2649 00000100:00080000:0.0:1734018386.806549:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018386.806552:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018386.806557:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018386.806565:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332968576/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018386.806573:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018386.806577:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018386.806578:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018386.806581:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2649 00000100:00080000:0.0:1734018386.806584:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018386.806588:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018386.806592:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018386.806597:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332968320:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018386.806600:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332968704/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018386.806609:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018386.806612:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018386.806614:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018386.806617:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2649 00000100:00080000:0.0:1734018386.806620:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018386.806623:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018386.806627:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018386.806635:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332968832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018386.806643:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018386.806646:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018386.806647:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018386.806651:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2649 00000100:00080000:0.0:1734018386.806654:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018386.806657:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018386.806661:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018386.806668:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018386.806670:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332968960/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018386.806678:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018386.806681:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018386.806683:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018386.806686:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2649 00000100:00080000:0.0:1734018386.806689:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018386.806689:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018386/real 1734018386] req@ffff8800a705fb80 x1818247332968320/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018386.806693:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018386.806697:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018386.806704:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332969088/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018386.806709:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332968320/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018386.806719:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018386.806723:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018386.806727:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332968320:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018386.806734:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332968448:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018386.806751:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018386.806757:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018386/real 1734018386] req@ffff8800a705ea00 x1818247332968448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018386.806768:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332968448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018386.806777:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018386.806779:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018386.806782:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332968448:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018386.806788:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332968576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018386.806801:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018386.806807:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018386/real 1734018386] req@ffff8800a705ed80 x1818247332968576/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018386.806817:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332968576/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018386.806825:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018386.806827:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018386.806830:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332968576:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018386.806855:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332968704:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018386.806867:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018386.806873:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018386/real 1734018386] req@ffff8800a705c380 x1818247332968704/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018386.806883:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332968704/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018386.806891:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018386.806893:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018386.806896:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332968704:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018386.806901:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332968832:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018386.806913:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018386.806918:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018386/real 1734018386] req@ffff8800a705d180 x1818247332968832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018386.806928:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332968832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018386.806935:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018386.806937:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018386.806941:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332968832:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018386.806947:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332968960:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018386.806959:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018386.806965:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018386/real 1734018386] req@ffff8800a705dc00 x1818247332968960/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018386.806975:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332968960/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018386.806982:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018386.806984:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018386.806987:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332968960:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018386.806993:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332969088:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018386.807004:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018386.807010:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018386/real 1734018386] req@ffff8800a705f100 x1818247332969088/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018386.807020:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332969088/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018386.807027:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018386.807029:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018386.807032:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332969088:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018387.480468:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018388.482483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018389.484503:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018390.486509:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018391.488536:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018391.814400:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018391.814412:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018391.814418:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018391.814433:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2654 00000100:00080000:0.0:1734018391.814445:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018391.814454:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018391.814503:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332969216/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018391.814541:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018391.814545:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018391.814548:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018391.814553:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2654 00000100:00080000:0.0:1734018391.814556:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018391.814560:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018391.814572:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018391.814584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332969344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018391.814595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018391.814599:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018391.814600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018391.814604:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2654 00000100:00080000:0.0:1734018391.814607:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018391.814610:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018391.814615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018391.814627:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332969472/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018391.814636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018391.814639:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018391.814640:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018391.814644:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2654 00000100:00080000:0.0:1734018391.814647:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018391.814650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018391.814655:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018391.814663:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332969600/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018391.814673:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018391.814677:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018391.814678:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018391.814682:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2654 00000100:00100000:1.0:1734018391.814682:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332969216:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018391.814686:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018391.814690:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018391.814694:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018391.814704:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332969728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018391.814713:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018391.814716:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018391.814717:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018391.814721:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2654 00000100:00080000:0.0:1734018391.814724:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018391.814727:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018391.814731:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018391.814741:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332969856/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018391.814749:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018391.814752:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018391.814753:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018391.814757:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2654 00000100:00080000:0.0:1734018391.814760:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018391.814760:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018391.814764:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018391.814769:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018391.814777:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332969984/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018391.814786:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018391/real 1734018391] req@ffff8800a705c000 x1818247332969216/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018391.814807:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332969216/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018391.814819:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018391.814823:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018391.814827:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332969216:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018391.814835:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332969344:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018391.814854:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018391.814860:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018391/real 1734018391] req@ffff8800a705e300 x1818247332969344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018391.814871:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332969344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018391.814879:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018391.814882:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018391.814886:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332969344:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018391.814892:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332969472:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018391.814906:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018391.814912:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018391/real 1734018391] req@ffff8800a705f100 x1818247332969472/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018391.814922:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332969472/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018391.814930:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018391.814933:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018391.814936:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332969472:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018391.814963:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332969600:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018391.814976:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018391.814982:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018391/real 1734018391] req@ffff8800a705dc00 x1818247332969600/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018391.814993:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332969600/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018391.815000:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018391.815003:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018391.815006:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332969600:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018391.815011:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332969728:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018391.815024:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018391.815030:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018391/real 1734018391] req@ffff8800a705d180 x1818247332969728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018391.815039:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332969728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018391.815047:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018391.815049:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018391.815052:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332969728:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018391.815058:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332969856:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018391.815074:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018391.815080:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018391/real 1734018391] req@ffff8800a705c380 x1818247332969856/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018391.815090:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332969856/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018391.815097:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018391.815099:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018391.815102:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332969856:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018391.815108:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332969984:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018391.815120:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018391.815125:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018391/real 1734018391] req@ffff8800a705ed80 x1818247332969984/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018391.815135:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332969984/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018391.815142:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018391.815145:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018391.815148:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332969984:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018392.490528:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018393.492448:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018394.494470:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018395.496511:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018396.498500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018396.822361:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018396.822371:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018396.822376:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018396.822386:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2659 00000100:00080000:0.0:1734018396.822396:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018396.822404:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018396.822441:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332970112/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018396.822475:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018396.822480:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018396.822482:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018396.822486:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2659 00000100:00080000:0.0:1734018396.822489:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018396.822493:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018396.822503:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018396.822514:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332970240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018396.822524:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018396.822527:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018396.822529:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018396.822533:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2659 00000100:00080000:0.0:1734018396.822535:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018396.822539:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018396.822544:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018396.822553:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332970368/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018396.822561:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018396.822564:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018396.822566:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018396.822569:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2659 00000100:00080000:0.0:1734018396.822572:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018396.822575:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018396.822579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018396.822582:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332970112:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018396.822587:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332970496/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018396.822595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018396.822599:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018396.822600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018396.822603:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2659 00000100:00080000:0.0:1734018396.822606:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018396.822609:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018396.822613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018396.822621:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332970624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018396.822630:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018396.822633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018396.822635:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018396.822639:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2659 00000100:00080000:0.0:1734018396.822641:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018396.822645:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018396.822649:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018396.822650:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018396.822657:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332970752/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018396.822665:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018396.822668:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018396.822669:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018396.822672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2659 00000100:00100000:1.0:1734018396.822673:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018396/real 1734018396] req@ffff8800a705ea00 x1818247332970112/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018396.822675:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018396.822678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018396.822682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018396.822691:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332970880/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018396.822701:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332970112/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018396.822711:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018396.822715:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018396.822719:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332970112:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018396.822727:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332970240:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018396.822744:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018396.822751:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018396/real 1734018396] req@ffff8800a705fb80 x1818247332970240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018396.822762:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332970240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018396.822769:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018396.822772:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018396.822775:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332970240:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018396.822781:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332970368:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018396.822794:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018396.822800:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018396/real 1734018396] req@ffff8800a705f800 x1818247332970368/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018396.822810:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332970368/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018396.822817:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018396.822821:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018396.822824:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332970368:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018396.822848:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332970496:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018396.822861:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018396.822867:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018396/real 1734018396] req@ffff8800a705d880 x1818247332970496/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018396.822877:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332970496/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018396.822885:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018396.822887:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018396.822890:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332970496:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018396.822896:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332970624:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018396.822907:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018396.822913:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018396/real 1734018396] req@ffff8800a705ca80 x1818247332970624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018396.822923:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332970624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018396.822930:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018396.822932:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018396.822935:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332970624:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018396.822941:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332970752:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018396.822956:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018396.822962:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018396/real 1734018396] req@ffff8800a705df80 x1818247332970752/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018396.822973:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332970752/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018396.822980:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018396.822982:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018396.822985:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332970752:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018396.822990:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332970880:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018396.823001:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018396.823007:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018396/real 1734018396] req@ffff8800a705c700 x1818247332970880/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018396.823016:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332970880/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018396.823024:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018396.823026:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018396.823029:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332970880:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018397.500478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018398.502457:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018399.504469:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018400.506489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018401.508484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018401.830364:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018401.830377:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018401.830383:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018401.830397:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2664 00000100:00080000:0.0:1734018401.830410:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018401.830421:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018401.830468:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332971008/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018401.830515:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018401.830520:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018401.830522:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018401.830527:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2664 00000100:00080000:0.0:1734018401.830531:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018401.830535:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018401.830547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018401.830559:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332971136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018401.830569:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018401.830573:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018401.830575:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018401.830579:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2664 00000100:00080000:0.0:1734018401.830581:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018401.830585:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018401.830589:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018401.830600:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332971264/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018401.830609:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018401.830612:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018401.830613:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018401.830617:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2664 00000100:00080000:0.0:1734018401.830619:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018401.830624:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018401.830629:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018401.830638:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332971392/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018401.830647:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332971008:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018401.830648:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018401.830651:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018401.830653:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018401.830657:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2664 00000100:00080000:0.0:1734018401.830659:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018401.830663:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018401.830668:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018401.830676:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332971520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018401.830684:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018401.830688:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018401.830689:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018401.830693:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2664 00000100:00080000:0.0:1734018401.830707:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018401.830711:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018401.830717:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018401.830721:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018401.830725:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332971648/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018401.830734:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018401.830739:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018401.830740:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018401.830744:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2664 00000100:00100000:1.0:1734018401.830744:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018401/real 1734018401] req@ffff8800a705d500 x1818247332971008/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018401.830748:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018401.830751:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018401.830755:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018401.830764:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332971776/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018401.830772:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332971008/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018401.830783:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018401.830787:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018401.830792:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332971008:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018401.830800:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332971136:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018401.830819:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018401.830825:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018401/real 1734018401] req@ffff8800a705e680 x1818247332971136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018401.830836:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332971136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018401.830844:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018401.830847:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018401.830850:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332971136:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018401.830857:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332971264:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018401.830870:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018401.830876:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018401/real 1734018401] req@ffff8800a705c700 x1818247332971264/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018401.830886:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332971264/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018401.830893:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018401.830897:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018401.830900:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332971264:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018401.830926:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332971392:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018401.830939:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018401.830946:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018401/real 1734018401] req@ffff8800a705df80 x1818247332971392/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018401.830956:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332971392/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018401.830964:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018401.830966:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018401.830969:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332971392:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018401.830975:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332971520:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018401.830987:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018401.830993:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018401/real 1734018401] req@ffff8800a705ca80 x1818247332971520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018401.831003:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332971520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018401.831010:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018401.831012:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018401.831015:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332971520:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018401.831022:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332971648:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018401.831038:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018401.831044:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018401/real 1734018401] req@ffff8800a705d880 x1818247332971648/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018401.831054:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332971648/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018401.831061:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018401.831064:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018401.831067:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332971648:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018401.831072:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332971776:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018401.831084:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018401.831090:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018401/real 1734018401] req@ffff8800a705f800 x1818247332971776/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018401.831100:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332971776/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018401.831107:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018401.831109:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018401.831112:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332971776:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018402.510491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018403.512483:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018404.514486:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018405.516448:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018406.518522:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018406.838340:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018406.838353:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018406.838359:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018406.838373:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2669 00000100:00080000:0.0:1734018406.838385:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018406.838395:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018406.838442:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332971904/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018406.838481:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018406.838485:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018406.838487:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018406.838493:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2669 00000100:00080000:0.0:1734018406.838496:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018406.838500:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018406.838512:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018406.838525:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332972032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018406.838536:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018406.838540:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018406.838542:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018406.838545:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2669 00000100:00080000:0.0:1734018406.838548:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018406.838552:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018406.838556:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018406.838565:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332972160/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018406.838574:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018406.838578:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018406.838579:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018406.838583:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2669 00000100:00080000:0.0:1734018406.838585:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018406.838589:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018406.838593:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018406.838602:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332972288/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018406.838608:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332971904:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018406.838612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018406.838616:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018406.838617:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018406.838622:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2669 00000100:00080000:0.0:1734018406.838625:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018406.838629:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018406.838633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018406.838642:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332972416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018406.838650:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018406.838653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018406.838655:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018406.838659:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2669 00000100:00080000:0.0:1734018406.838672:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018406.838676:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018406.838681:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018406.838684:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018406.838689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332972544/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018406.838697:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018406.838701:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018406.838702:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018406.838706:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2669 00000100:00080000:0.0:1734018406.838709:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018406.838710:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018406/real 1734018406] req@ffff8800a705fb80 x1818247332971904/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018406.838713:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018406.838718:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018406.838728:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332972672/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018406.838737:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332971904/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018406.838749:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018406.838752:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018406.838757:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332971904:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018406.838765:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332972032:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018406.838784:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018406.838790:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018406/real 1734018406] req@ffff8800a705ea00 x1818247332972032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018406.838801:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332972032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018406.838809:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018406.838812:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018406.838816:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332972032:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018406.838822:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332972160:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018406.838836:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018406.838842:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018406/real 1734018406] req@ffff8800a705ed80 x1818247332972160/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018406.838852:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332972160/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018406.838860:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018406.838863:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018406.838866:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332972160:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018406.838892:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332972288:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018406.838905:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018406.838911:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018406/real 1734018406] req@ffff8800a705c380 x1818247332972288/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018406.838922:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332972288/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018406.838929:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018406.838932:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018406.838935:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332972288:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018406.838941:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332972416:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018406.838953:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018406.838959:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018406/real 1734018406] req@ffff8800a705d180 x1818247332972416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018406.838969:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332972416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018406.838976:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018406.838979:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018406.838982:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332972416:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018406.838988:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332972544:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018406.839004:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018406.839010:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018406/real 1734018406] req@ffff8800a705dc00 x1818247332972544/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018406.839020:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332972544/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018406.839027:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018406.839030:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018406.839032:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332972544:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018406.839038:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332972672:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018406.839050:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018406.839056:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018406/real 1734018406] req@ffff8800a705f100 x1818247332972672/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018406.839124:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332972672/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018406.839133:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018406.839136:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018406.839139:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332972672:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018407.520479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018408.522436:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018409.524477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018410.526499:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018411.528499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018411.846354:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018411.846367:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018411.846373:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018411.846386:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2674 00000100:00080000:0.0:1734018411.846398:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018411.846407:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018411.846456:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332972800/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018411.846494:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018411.846499:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018411.846501:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018411.846507:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2674 00000100:00080000:0.0:1734018411.846510:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018411.846514:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018411.846525:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018411.846537:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332972928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018411.846547:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018411.846551:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018411.846552:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018411.846557:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2674 00000100:00080000:0.0:1734018411.846560:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018411.846563:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018411.846569:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018411.846581:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332973056/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018411.846590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018411.846593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018411.846595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018411.846598:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2674 00000100:00080000:0.0:1734018411.846601:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018411.846604:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018411.846609:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018411.846617:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332973184/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018411.846627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018411.846628:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332972800:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018411.846631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018411.846633:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018411.846637:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2674 00000100:00080000:0.0:1734018411.846640:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018411.846643:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018411.846648:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018411.846657:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332973312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018411.846665:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018411.846668:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018411.846670:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018411.846674:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2674 00000100:00080000:0.0:1734018411.846691:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018411.846695:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018411.846700:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018411.846706:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018411.846709:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332973440/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018411.846718:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018411.846721:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018411.846722:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018411.846726:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2674 00000100:00080000:0.0:1734018411.846729:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018411.846731:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018411/real 1734018411] req@ffff8800a705e300 x1818247332972800/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018411.846733:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018411.846737:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018411.846745:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332973568/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018411.846760:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332972800/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018411.846772:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018411.846775:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018411.846780:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332972800:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018411.846788:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332972928:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018411.846806:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018411.846813:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018411/real 1734018411] req@ffff8800a705c000 x1818247332972928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018411.846824:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332972928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018411.846832:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018411.846835:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018411.846838:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332972928:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018411.846845:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332973056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018411.846859:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018411.846865:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018411/real 1734018411] req@ffff8800a705f100 x1818247332973056/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018411.846875:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332973056/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018411.846882:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018411.846886:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018411.846889:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332973056:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018411.846917:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332973184:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018411.846930:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018411.846936:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018411/real 1734018411] req@ffff8800a705dc00 x1818247332973184/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018411.846947:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332973184/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018411.846955:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018411.846957:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018411.846960:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332973184:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018411.846966:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332973312:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018411.847032:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018411.847041:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018411/real 1734018411] req@ffff8800a705d180 x1818247332973312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018411.847052:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332973312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018411.847060:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018411.847062:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018411.847065:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332973312:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018411.847071:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332973440:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018411.847092:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018411.847098:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018411/real 1734018411] req@ffff8800a705c380 x1818247332973440/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018411.847109:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332973440/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018411.847116:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018411.847118:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018411.847121:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332973440:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018411.847127:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332973568:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018411.847139:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018411.847145:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018411/real 1734018411] req@ffff8800a705ed80 x1818247332973568/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018411.847155:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332973568/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018411.847162:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018411.847164:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018411.847167:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332973568:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018412.530479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018413.532480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018414.534508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018415.536491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018416.538491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018416.854372:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018416.854382:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018416.854386:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018416.854397:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2679 00000100:00080000:0.0:1734018416.854409:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018416.854416:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018416.854452:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332973696/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018416.854495:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018416.854499:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018416.854501:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018416.854506:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2679 00000100:00080000:0.0:1734018416.854509:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018416.854513:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018416.854522:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018416.854533:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332973824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018416.854542:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018416.854546:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018416.854547:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018416.854551:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2679 00000100:00080000:0.0:1734018416.854554:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018416.854558:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018416.854562:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018416.854570:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332973952/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018416.854579:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018416.854582:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018416.854583:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018416.854587:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2679 00000100:00080000:0.0:1734018416.854589:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018416.854593:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018416.854598:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018416.854605:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332974080/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018416.854614:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018416.854616:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332973696:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018416.854617:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018416.854619:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018416.854622:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2679 00000100:00080000:0.0:1734018416.854625:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018416.854628:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018416.854632:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018416.854640:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332974208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018416.854648:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018416.854651:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018416.854652:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018416.854656:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2679 00000100:00080000:0.0:1734018416.854670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018416.854673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018416.854678:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018416.854685:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018416.854686:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332974336/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018416.854694:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018416.854698:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018416.854700:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018416.854703:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2679 00000100:00080000:0.0:1734018416.854706:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018416.854706:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018416/real 1734018416] req@ffff8800a705ea00 x1818247332973696/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018416.854709:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018416.854714:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018416.854723:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332974464/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018416.854732:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332973696/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018416.854743:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018416.854746:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018416.854751:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332973696:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018416.854758:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332973824:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018416.854775:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018416.854781:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018416/real 1734018416] req@ffff8800a705fb80 x1818247332973824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018416.854792:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332973824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018416.854800:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018416.854803:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018416.854806:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332973824:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018416.854811:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332973952:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018416.854825:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018416.854830:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018416/real 1734018416] req@ffff8800a705f800 x1818247332973952/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018416.854840:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332973952/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018416.854848:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018416.854851:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018416.854854:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332973952:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018416.854880:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332974080:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018416.854944:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018416.854954:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018416/real 1734018416] req@ffff8800a705d880 x1818247332974080/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018416.854966:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332974080/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018416.854974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018416.854976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018416.854979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332974080:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018416.854984:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332974208:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018416.855001:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018416.855007:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018416/real 1734018416] req@ffff8800a705ca80 x1818247332974208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018416.855018:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332974208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018416.855026:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018416.855028:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018416.855031:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332974208:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018416.855036:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332974336:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018416.855051:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018416.855057:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018416/real 1734018416] req@ffff8800a705df80 x1818247332974336/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018416.855067:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332974336/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018416.855074:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018416.855076:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018416.855079:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332974336:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018416.855084:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332974464:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018416.855096:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018416.855102:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018416/real 1734018416] req@ffff8800a705c700 x1818247332974464/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018416.855112:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332974464/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018416.855119:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018416.855121:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018416.855124:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332974464:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018417.540481:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018418.542513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018419.544482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018420.546528:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018421.548458:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018421.862330:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018421.862338:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018421.862342:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018421.862351:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2684 00000100:00080000:0.0:1734018421.862360:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018421.862366:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018421.862400:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332974592/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018421.862424:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018421.862427:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018421.862428:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018421.862431:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2684 00000100:00080000:0.0:1734018421.862433:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018421.862435:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018421.862443:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018421.862450:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332974720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018421.862456:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018421.862458:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018421.862459:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018421.862461:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2684 00000100:00080000:0.0:1734018421.862463:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018421.862465:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018421.862467:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018421.862474:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332974848/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018421.862478:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018421.862480:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018421.862481:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018421.862483:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2684 00000100:00080000:0.0:1734018421.862484:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018421.862486:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018421.862489:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018421.862494:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332974976/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018421.862499:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018421.862501:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018421.862502:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018421.862504:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2684 00000100:00080000:0.0:1734018421.862506:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018421.862508:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018421.862510:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018421.862515:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332975104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018421.862520:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018421.862522:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018421.862523:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018421.862525:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2684 00000100:00080000:0.0:1734018421.862526:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018421.862528:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018421.862529:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332974592:192.168.201.152@tcp:250:kworker.0 00000008:00100000:0.0:1734018421.862531:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018421.862535:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332975232/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018421.862540:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018421.862542:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018421.862542:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018421.862544:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2684 00000100:00080000:0.0:1734018421.862545:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018421.862548:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018421.862550:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018421.862555:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332975360/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018421.862586:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018421.862604:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018421/real 1734018421] req@ffff8800a705e680 x1818247332974592/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018421.862622:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332974592/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018421.862629:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018421.862631:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018421.862634:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332974592:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018421.862638:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332974720:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018421.862650:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018421.862654:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018421/real 1734018421] req@ffff8800a705d500 x1818247332974720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018421.862661:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332974720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018421.862665:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018421.862667:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018421.862669:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332974720:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018421.862672:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332974848:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018421.862681:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018421.862685:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018421/real 1734018421] req@ffff8800a705c700 x1818247332974848/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018421.862690:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332974848/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018421.862695:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018421.862697:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018421.862698:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332974848:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018421.862702:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332974976:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018421.862710:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018421.862713:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018421/real 1734018421] req@ffff8800a705df80 x1818247332974976/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018421.862719:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332974976/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018421.862723:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018421.862725:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018421.862727:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332974976:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018421.862746:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332975104:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018421.862754:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018421.862757:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018421/real 1734018421] req@ffff8800a705ca80 x1818247332975104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018421.862763:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332975104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018421.862768:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018421.862769:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018421.862771:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332975104:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018421.862774:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332975232:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018421.862784:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018421.862788:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018421/real 1734018421] req@ffff8800a705d880 x1818247332975232/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018421.862794:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332975232/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018421.862798:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018421.862799:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018421.862801:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332975232:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018421.862804:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332975360:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018421.862834:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018421.862839:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018421/real 1734018421] req@ffff8800a705f800 x1818247332975360/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018421.862861:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332975360/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018421.862865:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018421.862866:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018421.862868:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332975360:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018422.550512:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018423.552541:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018424.554485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018425.556470:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018426.558477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018426.870354:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018426.870367:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018426.870373:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018426.870388:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2689 00000100:00080000:0.0:1734018426.870400:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018426.870410:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018426.870458:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332975488/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018426.870497:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018426.870502:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018426.870504:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018426.870509:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2689 00000100:00080000:0.0:1734018426.870512:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018426.870517:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018426.870529:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018426.870541:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332975616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018426.870551:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018426.870555:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018426.870557:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018426.870561:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2689 00000100:00080000:0.0:1734018426.870564:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018426.870567:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018426.870573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018426.870582:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332975744/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018426.870592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018426.870595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018426.870596:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018426.870600:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2689 00000100:00080000:0.0:1734018426.870602:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018426.870606:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018426.870611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018426.870619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332975872/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018426.870628:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018426.870631:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332975488:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018426.870632:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018426.870634:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018426.870638:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2689 00000100:00080000:0.0:1734018426.870640:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018426.870644:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018426.870649:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018426.870657:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332976000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018426.870666:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018426.870670:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018426.870672:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018426.870676:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2689 00000100:00080000:0.0:1734018426.870694:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018426.870698:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018426.870704:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018426.870708:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018426.870713:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332976128/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018426.870721:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018426.870760:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018426.870761:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018426.870766:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2689 00000100:00080000:0.0:1734018426.870770:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018426.870774:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018426.870778:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018426/real 1734018426] req@ffff8800a705fb80 x1818247332975488/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018426.870779:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018426.870792:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332976256/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018426.870806:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332975488/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018426.870818:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018426.870822:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018426.870827:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332975488:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018426.870836:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332975616:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018426.870857:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018426.870864:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018426/real 1734018426] req@ffff8800a705ea00 x1818247332975616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018426.870875:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332975616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018426.870884:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018426.870887:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018426.870890:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332975616:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018426.870897:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332975744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018426.870910:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018426.870917:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018426/real 1734018426] req@ffff8800a705ed80 x1818247332975744/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018426.870927:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332975744/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018426.870934:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018426.870937:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018426.870940:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332975744:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018426.870968:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332975872:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018426.870981:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018426.870987:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018426/real 1734018426] req@ffff8800a705c380 x1818247332975872/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018426.870998:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332975872/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018426.871005:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018426.871007:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018426.871010:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332975872:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018426.871016:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332976000:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018426.871029:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018426.871035:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018426/real 1734018426] req@ffff8800a705d180 x1818247332976000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018426.871045:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332976000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018426.871052:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018426.871055:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018426.871058:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332976000:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018426.871064:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332976128:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018426.871079:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018426.871085:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018426/real 1734018426] req@ffff8800a705dc00 x1818247332976128/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018426.871096:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332976128/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018426.871103:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018426.871105:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018426.871108:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332976128:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018426.871114:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332976256:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018426.871126:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018426.871132:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018426/real 1734018426] req@ffff8800a705f100 x1818247332976256/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018426.871143:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332976256/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018426.871150:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018426.871152:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018426.871155:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332976256:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018427.560418:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018428.562469:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018429.564482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018430.566504:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018431.568494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018431.878357:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018431.878367:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018431.878371:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018431.878382:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2694 00000100:00080000:0.0:1734018431.878391:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018431.878399:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018431.878433:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332976384/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018431.878474:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018431.878479:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018431.878480:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018431.878485:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2694 00000100:00080000:0.0:1734018431.878488:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018431.878492:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018431.878502:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018431.878513:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332976512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018431.878522:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018431.878526:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018431.878527:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018431.878531:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2694 00000100:00080000:0.0:1734018431.878534:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018431.878537:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018431.878542:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018431.878552:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332976640/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018431.878560:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018431.878564:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018431.878565:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018431.878568:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2694 00000100:00080000:0.0:1734018431.878571:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018431.878575:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018431.878579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018431.878582:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332976384:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018431.878588:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332976768/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018431.878596:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018431.878599:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018431.878601:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018431.878604:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2694 00000100:00080000:0.0:1734018431.878607:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018431.878610:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018431.878615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018431.878622:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332976896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018431.878630:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018431.878633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018431.878635:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018431.878671:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2694 00000100:00080000:0.0:1734018431.878674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018431.878678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018431.878683:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018431.878691:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018431.878692:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332977024/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018431.878701:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018431.878705:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018431.878707:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018431.878710:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2694 00000100:00080000:0.0:1734018431.878713:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018431.878715:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018431/real 1734018431] req@ffff8800a705c000 x1818247332976384/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018431.878716:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018431.878720:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018431.878729:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332977152/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018431.878742:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332976384/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018431.878753:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018431.878756:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018431.878761:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332976384:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018431.878768:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332976512:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018431.878789:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018431.878796:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018431/real 1734018431] req@ffff8800a705e300 x1818247332976512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018431.878806:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332976512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018431.878815:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018431.878817:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018431.878821:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332976512:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018431.878826:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332976640:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018431.878840:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018431.878845:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018431/real 1734018431] req@ffff8800a705f100 x1818247332976640/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018431.878855:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332976640/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018431.878863:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018431.878866:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018431.878869:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332976640:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018431.878893:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332976768:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018431.878906:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018431.878912:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018431/real 1734018431] req@ffff8800a705dc00 x1818247332976768/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018431.878922:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332976768/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018431.878930:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018431.878932:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018431.878935:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332976768:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018431.878940:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332976896:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018431.878952:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018431.878957:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018431/real 1734018431] req@ffff8800a705d180 x1818247332976896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018431.878967:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332976896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018431.878974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018431.878977:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018431.878980:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332976896:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018431.878986:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332977024:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018431.879001:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018431.879007:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018431/real 1734018431] req@ffff8800a705c380 x1818247332977024/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018431.879017:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332977024/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018431.879024:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018431.879026:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018431.879029:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332977024:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018431.879034:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332977152:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018431.879045:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018431.879051:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018431/real 1734018431] req@ffff8800a705ed80 x1818247332977152/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018431.879061:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332977152/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018431.879068:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018431.879070:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018431.879073:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332977152:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018432.570447:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018433.572415:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018434.574447:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018435.576486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018436.578511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018436.886343:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018436.886352:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018436.886357:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018436.886368:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2699 00000100:00080000:0.0:1734018436.886379:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018436.886387:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018436.886424:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332977280/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018436.886458:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018436.886462:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018436.886465:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018436.886469:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2699 00000100:00080000:0.0:1734018436.886472:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018436.886476:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018436.886486:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018436.886496:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332977408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018436.886507:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018436.886511:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018436.886512:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018436.886516:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2699 00000100:00080000:0.0:1734018436.886519:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018436.886522:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018436.886527:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018436.886535:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332977536/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018436.886543:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018436.886546:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018436.886548:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018436.886551:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2699 00000100:00080000:0.0:1734018436.886582:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018436.886586:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018436.886591:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018436.886595:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332977280:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018436.886600:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332977664/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018436.886609:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018436.886613:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018436.886614:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018436.886617:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2699 00000100:00080000:0.0:1734018436.886621:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018436.886625:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018436.886629:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018436.886637:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332977792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018436.886645:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018436.886648:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018436.886650:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018436.886653:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2699 00000100:00080000:0.0:1734018436.886656:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018436.886659:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018436.886664:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018436.886665:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018436.886671:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332977920/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018436.886679:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018436.886683:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018436.886684:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018436.886687:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2699 00000100:00100000:1.0:1734018436.886688:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018436/real 1734018436] req@ffff8800a705ea00 x1818247332977280/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018436.886690:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018436.886693:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018436.886697:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018436.886707:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332978048/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018436.886715:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332977280/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018436.886725:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018436.886729:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018436.886734:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332977280:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018436.886741:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332977408:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018436.886757:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018436.886764:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018436/real 1734018436] req@ffff8800a705fb80 x1818247332977408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018436.886774:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332977408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018436.886782:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018436.886785:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018436.886789:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332977408:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018436.886794:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332977536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018436.886808:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018436.886814:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018436/real 1734018436] req@ffff8800a705f800 x1818247332977536/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018436.886824:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332977536/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018436.886831:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018436.886834:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018436.886838:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332977536:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018436.886862:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332977664:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018436.886875:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018436.886881:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018436/real 1734018436] req@ffff8800a705d880 x1818247332977664/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018436.886892:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332977664/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018436.886899:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018436.886901:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018436.886904:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332977664:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018436.886909:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332977792:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018436.886921:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018436.886927:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018436/real 1734018436] req@ffff8800a705ca80 x1818247332977792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018436.886936:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332977792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018436.886944:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018436.886946:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018436.886950:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332977792:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018436.886955:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332977920:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018436.886970:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018436.886976:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018436/real 1734018436] req@ffff8800a705df80 x1818247332977920/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018436.886986:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332977920/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018436.886993:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018436.886995:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018436.886998:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332977920:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018436.887003:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332978048:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018436.887015:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018436.887020:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018436/real 1734018436] req@ffff8800a705c700 x1818247332978048/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018436.887030:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332978048/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018436.887037:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018436.887040:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018436.887042:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332978048:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018437.580472:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018438.582486:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018439.584506:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018440.586550:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018441.588543:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018441.894341:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018441.894351:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018441.894355:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018441.894366:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2704 00000100:00080000:0.0:1734018441.894376:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018441.894384:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018441.894420:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332978176/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018441.894453:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018441.894457:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018441.894459:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018441.894464:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2704 00000100:00080000:0.0:1734018441.894508:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018441.894514:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018441.894524:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018441.894536:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332978304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018441.894547:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018441.894551:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018441.894552:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018441.894556:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2704 00000100:00080000:0.0:1734018441.894560:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018441.894560:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332978176:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018441.894563:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018441.894569:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018441.894579:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332978432/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018441.894589:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018441.894592:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018441.894594:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018441.894597:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2704 00000100:00080000:0.0:1734018441.894600:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018441.894603:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018441.894608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018441.894615:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332978560/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018441.894623:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018441.894627:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018441.894628:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734018441.894630:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018441.894632:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2704 00000100:00080000:0.0:1734018441.894634:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018441.894638:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018441.894642:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018441.894649:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332978688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018441.894652:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018441/real 1734018441] req@ffff8800a705d500 x1818247332978176/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018441.894657:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018441.894660:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018441.894662:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018441.894666:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2704 00000100:00080000:0.0:1734018441.894669:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018441.894673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018441.894677:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018441.894679:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332978176/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018441.894684:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332978816/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018441.894689:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018441.894692:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1734018441.894692:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:0.0:1734018441.894695:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018441.894696:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018441.894697:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332978176:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018441.894700:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2704 00000100:00080000:0.0:1734018441.894702:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018441.894705:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018441.894709:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018441.894717:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332978944/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018441.894718:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332978304:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018441.894736:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018441.894743:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018441/real 1734018441] req@ffff8800a705e680 x1818247332978304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018441.894754:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332978304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018441.894762:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018441.894764:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018441.894768:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332978304:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018441.894773:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332978432:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018441.894787:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018441.894793:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018441/real 1734018441] req@ffff8800a705c700 x1818247332978432/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018441.894803:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332978432/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018441.894810:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018441.894814:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018441.894817:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332978432:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018441.894823:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332978560:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018441.894835:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018441.894841:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018441/real 1734018441] req@ffff8800a705df80 x1818247332978560/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018441.894850:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332978560/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018441.894858:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018441.894860:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018441.894864:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332978560:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018441.894869:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332978688:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018441.894881:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018441.894886:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018441/real 1734018441] req@ffff8800a705ca80 x1818247332978688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018441.894896:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332978688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018441.894904:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018441.894906:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018441.894909:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332978688:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018441.894915:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332978816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018441.894930:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018441.894936:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018441/real 1734018441] req@ffff8800a705d880 x1818247332978816/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018441.894946:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332978816/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018441.894953:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018441.894955:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018441.894958:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332978816:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018441.894975:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332978944:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018441.894987:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018441.894993:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018441/real 1734018441] req@ffff8800a705f800 x1818247332978944/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018441.895003:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332978944/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018441.895010:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018441.895012:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018441.895015:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332978944:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018442.590512:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018443.592491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018444.594498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018445.596482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018446.598511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018446.902364:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018446.902377:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018446.902410:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018446.902427:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2709 00000100:00080000:0.0:1734018446.902439:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018446.902449:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018446.902496:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332979072/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018446.902541:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018446.902546:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018446.902548:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018446.902553:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2709 00000100:00080000:0.0:1734018446.902556:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018446.902561:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018446.902573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018446.902585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332979200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018446.902595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018446.902599:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018446.902600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018446.902604:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2709 00000100:00080000:0.0:1734018446.902607:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018446.902610:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018446.902615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018446.902624:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332979328/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018446.902633:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018446.902636:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018446.902637:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018446.902641:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2709 00000100:00080000:0.0:1734018446.902643:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018446.902648:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018446.902653:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018446.902662:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332979456/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018446.902671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018446.902673:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332979072:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018446.902674:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018446.902676:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018446.902680:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2709 00000100:00080000:0.0:1734018446.902683:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018446.902686:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018446.902691:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018446.902700:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332979584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018446.902708:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018446.902711:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018446.902712:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018446.902717:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2709 00000100:00080000:0.0:1734018446.902719:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018446.902723:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018446.902728:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018446.902735:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332979712/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018446.902744:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018446.902748:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018446.902750:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018446.902754:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2709 00000100:00080000:0.0:1734018446.902757:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018446.902761:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018446.902765:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018446.902773:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332979840/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018446.902779:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332979200:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018446.902802:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332979328:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018446.902828:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332979456:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018446.902846:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332979584:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018446.902862:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332979712:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018446.902877:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332979840:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018447.600457:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018448.602459:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018449.604428:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734018449.908675:0:1956:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:3.0:1734018449.908730:0:1956:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734018449.908749:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734018449.908783:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734018449.908792:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734018449.908798:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734018449.908805:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734018449.908811:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734018449.908817:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734018449.908846:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018446/real 1734018449] req@ffff8800a705c380 x1818247332979456/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018449.908882:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332979456/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018449.908895:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018449.908899:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018449.908905:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332979456:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018449.908913:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018446/real 1734018449] req@ffff8800a705d180 x1818247332979584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018449.908925:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332979584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018449.908933:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018449.908936:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018449.908939:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332979584:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018449.908945:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018446/real 1734018449] req@ffff8800a705dc00 x1818247332979712/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018449.908956:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332979712/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018449.908964:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018449.908967:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018449.908970:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332979712:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018449.908976:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018446/real 1734018449] req@ffff8800a705f100 x1818247332979840/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018449.908987:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332979840/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018449.908994:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018449.908997:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018449.909000:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332979840:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018449.909006:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018446/real 1734018449] req@ffff8800a705fb80 x1818247332979072/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018449.909017:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332979072/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018449.909024:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018449.909027:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018449.909030:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332979072:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018449.909035:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018446/real 1734018449] req@ffff8800a705ea00 x1818247332979200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018449.909045:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332979200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018449.909060:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018449.909063:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018449.909066:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332979200:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018449.909072:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018446/real 1734018449] req@ffff8800a705ed80 x1818247332979328/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018449.909082:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332979328/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018449.909089:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018449.909092:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018449.909095:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332979328:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018450.606468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018451.608473:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018451.910387:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018451.910399:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018451.910403:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018451.910415:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2714 00000100:00080000:0.0:1734018451.910425:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018451.910434:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018451.910471:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332979968/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018451.910505:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018451.910509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018451.910511:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018451.910516:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2714 00000100:00080000:0.0:1734018451.910519:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018451.910523:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018451.910532:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018451.910543:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332980096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018451.910555:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018451.910558:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018451.910560:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018451.910563:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2714 00000100:00080000:0.0:1734018451.910566:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018451.910569:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018451.910574:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018451.910584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332980224/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018451.910592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018451.910595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018451.910597:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018451.910601:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2714 00000100:00080000:0.0:1734018451.910603:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018451.910606:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018451.910611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018451.910619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332980352/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018451.910622:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332979968:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018451.910628:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018451.910631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018451.910633:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018451.910636:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2714 00000100:00080000:0.0:1734018451.910640:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018451.910643:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018451.910647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018451.910655:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332980480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018451.910664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018451.910666:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018451.910668:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018451.910672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2714 00000100:00080000:0.0:1734018451.910676:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018451.910679:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018451.910683:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018451.910691:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332980608/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018451.910693:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018451.910699:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018451.910702:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018451.910703:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018451.910707:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2714 00000100:00080000:0.0:1734018451.910709:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018451.910713:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018451.910716:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018451/real 1734018451] req@ffff8800a705e300 x1818247332979968/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018451.910717:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018451.910725:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332980736/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018451.910735:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332979968/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018451.910745:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018451.910748:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018451.910753:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332979968:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018451.910760:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332980096:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018451.910777:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018451.910785:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018451/real 1734018451] req@ffff8800a705c000 x1818247332980096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018451.910797:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332980096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018451.910804:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018451.910807:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018451.910810:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332980096:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018451.910816:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332980224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018451.910830:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018451.910836:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018451/real 1734018451] req@ffff8800a705ed80 x1818247332980224/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018451.910846:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332980224/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018451.910853:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018451.910855:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018451.910858:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332980224:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018451.910884:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332980352:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018451.910896:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018451.910902:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018451/real 1734018451] req@ffff8800a705ea00 x1818247332980352/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018451.910913:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332980352/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018451.910928:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018451.910930:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018451.910933:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332980352:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018451.910939:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332980480:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018451.910951:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018451.910956:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018451/real 1734018451] req@ffff8800a705fb80 x1818247332980480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018451.910966:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332980480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018451.910974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018451.910976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018451.910979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332980480:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018451.910985:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332980608:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018451.910997:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018451.911002:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018451/real 1734018451] req@ffff8800a705f100 x1818247332980608/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018451.911012:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332980608/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018451.911019:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018451.911021:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018451.911024:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332980608:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018451.911030:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332980736:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018451.911043:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018451.911049:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018451/real 1734018451] req@ffff8800a705dc00 x1818247332980736/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018451.911058:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332980736/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018451.911065:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018451.911067:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018451.911070:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332980736:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018452.610529:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018453.612531:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018454.614492:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018455.616485:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018456.618522:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018456.918372:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018456.918385:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018456.918391:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018456.918405:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2719 00000100:00080000:0.0:1734018456.918416:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018456.918426:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018456.918476:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332980864/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018456.918514:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018456.918519:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018456.918521:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018456.918527:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2719 00000100:00080000:0.0:1734018456.918530:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018456.918534:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018456.918546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018456.918558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332980992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018456.918569:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018456.918573:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018456.918575:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018456.918579:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2719 00000100:00080000:0.0:1734018456.918582:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018456.918585:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018456.918592:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018456.918601:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332981120/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018456.918611:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018456.918614:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018456.918616:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018456.918619:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2719 00000100:00080000:0.0:1734018456.918622:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018456.918626:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018456.918630:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018456.918640:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332981248/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018456.918644:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332980864:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018456.918650:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018456.918653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018456.918655:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018456.918658:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2719 00000100:00080000:0.0:1734018456.918661:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018456.918665:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018456.918669:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018456.918678:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332981376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018456.918687:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018456.918690:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018456.918693:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018456.918698:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2719 00000100:00080000:0.0:1734018456.918711:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018456.918714:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018456.918719:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734018456.918720:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018456.918731:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332981504/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018456.918740:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018456.918744:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734018456.918744:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018456/real 1734018456] req@ffff8800a705d180 x1818247332980864/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018456.918746:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018456.918750:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2719 00000100:00080000:0.0:1734018456.918753:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018456.918756:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018456.918761:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018456.918764:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332980864/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018456.918769:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332981632/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018456.918776:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018456.918779:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018456.918784:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332980864:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018456.918793:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332980992:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018456.918811:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018456.918818:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018456/real 1734018456] req@ffff8800a705c380 x1818247332980992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018456.918838:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332980992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018456.918846:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018456.918849:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018456.918852:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332980992:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018456.918859:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332981120:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018456.918873:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018456.918879:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018456/real 1734018456] req@ffff8800a705f800 x1818247332981120/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018456.918889:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332981120/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018456.918896:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018456.918899:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018456.918902:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332981120:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018456.918929:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332981248:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018456.918942:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018456.918949:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018456/real 1734018456] req@ffff8800a705d880 x1818247332981248/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018456.918959:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332981248/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018456.918966:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018456.918970:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018456.918973:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332981248:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018456.918979:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332981376:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018456.918992:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018456.918998:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018456/real 1734018456] req@ffff8800a705ca80 x1818247332981376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018456.919008:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332981376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018456.919016:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018456.919018:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018456.919021:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332981376:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018456.919027:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332981504:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018456.919039:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018456.919045:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018456/real 1734018456] req@ffff8800a705df80 x1818247332981504/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018456.919055:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332981504/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018456.919062:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018456.919064:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018456.919068:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332981504:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018456.919074:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332981632:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018456.919089:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018456.919096:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018456/real 1734018456] req@ffff8800a705c700 x1818247332981632/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018456.919106:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332981632/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018456.919113:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018456.919115:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018456.919118:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332981632:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018457.620473:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018458.622467:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018459.624505:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018460.626513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018461.628471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018461.926314:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018461.926323:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018461.926328:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018461.926338:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2724 00000100:00080000:0.0:1734018461.926349:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018461.926357:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018461.926392:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332981760/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018461.926435:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018461.926439:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018461.926441:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018461.926445:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2724 00000100:00080000:0.0:1734018461.926449:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018461.926452:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018461.926462:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018461.926473:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332981888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018461.926483:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018461.926486:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018461.926488:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018461.926492:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2724 00000100:00080000:0.0:1734018461.926494:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018461.926498:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018461.926502:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018461.926512:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332982016/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018461.926521:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018461.926524:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018461.926525:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018461.926529:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2724 00000100:00080000:0.0:1734018461.926531:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018461.926535:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018461.926540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018461.926547:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332982144/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018461.926556:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018461.926556:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332981760:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018461.926559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018461.926561:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018461.926564:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2724 00000100:00080000:0.0:1734018461.926567:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018461.926571:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018461.926575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018461.926582:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332982272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018461.926591:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018461.926594:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018461.926595:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018461.926599:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2724 00000100:00080000:0.0:1734018461.926611:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018461.926614:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018461.926619:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018461.926626:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332982400/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018461.926626:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018461.926635:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018461.926639:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018461.926640:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018461.926644:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2724 00000100:00080000:0.0:1734018461.926647:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018461.926649:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018461/real 1734018461] req@ffff8800a705e680 x1818247332981760/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018461.926650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018461.926654:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018461.926662:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332982528/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018461.926668:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332981760/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018461.926679:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018461.926682:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018461.926687:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332981760:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018461.926694:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332981888:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018461.926711:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018461.926718:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018461/real 1734018461] req@ffff8800a705d500 x1818247332981888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018461.926729:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332981888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018461.926737:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018461.926740:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018461.926744:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332981888:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018461.926749:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332982016:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018461.926763:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018461.926769:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018461/real 1734018461] req@ffff8800a705c700 x1818247332982016/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018461.926778:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332982016/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018461.926786:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018461.926788:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018461.926791:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332982016:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018461.926816:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332982144:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018461.926829:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018461.926835:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018461/real 1734018461] req@ffff8800a705df80 x1818247332982144/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018461.926846:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332982144/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018461.926853:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018461.926855:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018461.926858:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332982144:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018461.926864:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332982272:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018461.926876:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018461.926882:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018461/real 1734018461] req@ffff8800a705ca80 x1818247332982272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018461.926893:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332982272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018461.926900:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018461.926903:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018461.926906:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332982272:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018461.926911:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332982400:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018461.926922:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018461.926928:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018461/real 1734018461] req@ffff8800a705d880 x1818247332982400/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018461.926938:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332982400/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018461.926945:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018461.926947:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018461.926950:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332982400:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018461.926956:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332982528:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018461.926967:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018461.926973:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018461/real 1734018461] req@ffff8800a705f800 x1818247332982528/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018461.926984:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332982528/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018461.926991:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018461.926994:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018461.926996:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332982528:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018462.630485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018463.632484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018464.634508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018465.636466:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018466.638519:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018466.934355:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018466.934366:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018466.934370:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018466.934381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2729 00000100:00080000:0.0:1734018466.934391:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018466.934399:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018466.934435:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332982656/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018466.934471:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018466.934475:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018466.934477:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018466.934482:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2729 00000100:00080000:0.0:1734018466.934485:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018466.934489:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018466.934498:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018466.934509:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332982784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018466.934520:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018466.934523:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018466.934525:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018466.934529:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2729 00000100:00080000:0.0:1734018466.934531:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018466.934534:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018466.934539:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018466.934547:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332982912/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018466.934556:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018466.934559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018466.934560:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018466.934563:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2729 00000100:00080000:0.0:1734018466.934566:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018466.934569:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018466.934573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018466.934581:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332983040/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018466.934588:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332982656:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018466.934590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018466.934593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018466.934594:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018466.934598:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2729 00000100:00080000:0.0:1734018466.934601:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018466.934604:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018466.934609:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018466.934616:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332983168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018466.934625:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018466.934627:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018466.934629:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018466.934632:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2729 00000100:00080000:0.0:1734018466.934644:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018466.934648:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018466.934652:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018466.934657:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018466.934662:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332983296/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018466.934670:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018466.934673:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018466.934674:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018466.934678:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2729 00000100:00100000:1.0:1734018466.934680:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018466/real 1734018466] req@ffff8800a705c380 x1818247332982656/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018466.934681:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018466.934684:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018466.934689:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018466.934696:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332983424/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018466.934700:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332982656/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018466.934710:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018466.934714:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018466.934718:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332982656:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018466.934725:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332982784:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018466.934742:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018466.934749:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018466/real 1734018466] req@ffff8800a705d180 x1818247332982784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018466.934760:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332982784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018466.934768:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018466.934771:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018466.934774:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332982784:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018466.934781:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332982912:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018466.934794:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018466.934800:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018466/real 1734018466] req@ffff8800a705dc00 x1818247332982912/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018466.934810:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332982912/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018466.934818:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018466.934820:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018466.934823:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332982912:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018466.934848:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332983040:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018466.934860:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018466.934865:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018466/real 1734018466] req@ffff8800a705f100 x1818247332983040/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018466.934876:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332983040/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018466.934883:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018466.934886:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018466.934889:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332983040:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018466.934894:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332983168:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018466.934909:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018466.934915:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018466/real 1734018466] req@ffff8800a705fb80 x1818247332983168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018466.934925:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332983168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018466.934932:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018466.934934:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018466.934937:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332983168:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018466.934943:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332983296:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018466.934954:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018466.934960:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018466/real 1734018466] req@ffff8800a705ea00 x1818247332983296/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018466.934970:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332983296/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018466.934977:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018466.934980:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018466.934983:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332983296:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018466.934988:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332983424:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018466.935000:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018466.935005:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018466/real 1734018466] req@ffff8800a705ed80 x1818247332983424/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018466.935022:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332983424/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018466.935029:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018466.935031:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018466.935034:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332983424:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018467.640454:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018468.642497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018469.644475:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018470.646503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018471.648511:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018471.942337:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018471.942349:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018471.942355:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018471.942369:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2734 00000100:00080000:0.0:1734018471.942381:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018471.942390:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018471.942440:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332983552/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018471.942479:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018471.942483:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018471.942485:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018471.942491:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2734 00000100:00080000:0.0:1734018471.942494:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018471.942498:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018471.942510:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018471.942522:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332983680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018471.942532:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018471.942536:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018471.942538:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018471.942542:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2734 00000100:00080000:0.0:1734018471.942545:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018471.942549:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018471.942555:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018471.942568:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332983808/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018471.942577:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018471.942580:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018471.942582:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018471.942585:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2734 00000100:00080000:0.0:1734018471.942588:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018471.942591:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018471.942596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018471.942603:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332983552:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018471.942605:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332983936/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018471.942615:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018471.942618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018471.942620:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018471.942624:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2734 00000100:00080000:0.0:1734018471.942626:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018471.942630:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018471.942635:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018471.942643:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332984064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018471.942652:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018471.942655:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018471.942657:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018471.942662:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2734 00000100:00080000:0.0:1734018471.942665:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018471.942669:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018471.942673:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018471.942680:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018471.942682:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332984192/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018471.942689:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018471.942693:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018471.942694:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018471.942698:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2734 00000100:00080000:0.0:1734018471.942701:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018471.942704:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018471.942705:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018471/real 1734018471] req@ffff8800a705c000 x1818247332983552/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018471.942709:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018471.942717:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332984320/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018471.942725:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332983552/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018471.942736:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018471.942740:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018471.942745:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332983552:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018471.942753:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332983680:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018471.942772:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018471.942779:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018471/real 1734018471] req@ffff8800a705e300 x1818247332983680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018471.942790:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332983680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018471.942798:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018471.942802:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018471.942806:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332983680:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018471.942814:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332983808:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018471.942838:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018471.942850:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018471/real 1734018471] req@ffff8800a705ed80 x1818247332983808/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018471.942868:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332983808/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018471.942880:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018471.942884:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018471.942889:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332983808:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018471.942928:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332983936:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018471.942956:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018471.943030:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018471/real 1734018471] req@ffff8800a705ea00 x1818247332983936/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018471.943043:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332983936/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018471.943052:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018471.943054:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018471.943058:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332983936:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018471.943065:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332984064:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018471.943090:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018471.943096:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018471/real 1734018471] req@ffff8800a705fb80 x1818247332984064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018471.943107:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332984064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018471.943114:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018471.943117:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018471.943120:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332984064:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018471.943125:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332984192:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018471.943138:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018471.943144:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018471/real 1734018471] req@ffff8800a705f100 x1818247332984192/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018471.943153:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332984192/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018471.943160:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018471.943163:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018471.943166:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332984192:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018471.943172:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332984320:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018471.943184:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018471.943190:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018471/real 1734018471] req@ffff8800a705dc00 x1818247332984320/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018471.943206:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332984320/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018471.943214:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018471.943272:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018471.943276:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332984320:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018472.650502:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018473.652487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018474.654531:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018475.656490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018476.658490:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018476.950338:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018476.950349:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018476.950355:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018476.950369:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2739 00000100:00080000:0.0:1734018476.950380:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018476.950390:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018476.950438:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332984448/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018476.950484:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018476.950489:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018476.950491:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018476.950496:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2739 00000100:00080000:0.0:1734018476.950500:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018476.950504:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018476.950515:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018476.950527:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332984576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018476.950537:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018476.950541:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018476.950542:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018476.950547:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2739 00000100:00080000:0.0:1734018476.950549:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018476.950553:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018476.950558:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018476.950567:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332984704/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018476.950575:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018476.950578:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018476.950580:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018476.950584:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2739 00000100:00080000:0.0:1734018476.950586:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018476.950591:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018476.950597:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018476.950606:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332984832/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018476.950606:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332984448:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018476.950616:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018476.950619:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018476.950621:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018476.950624:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2739 00000100:00080000:0.0:1734018476.950627:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018476.950630:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018476.950635:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018476.950644:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332984960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018476.950652:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018476.950655:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018476.950656:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018476.950661:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2739 00000100:00080000:0.0:1734018476.950666:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018476.950670:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018476.950675:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018476.950680:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018476.950683:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332985088/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018476.950691:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018476.950695:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018476.950697:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018476.950700:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2739 00000100:00080000:0.0:1734018476.950703:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018476.950706:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018476/real 1734018476] req@ffff8800a705d180 x1818247332984448/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018476.950707:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018476.950712:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018476.950721:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332985216/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018476.950726:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332984448/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018476.950738:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018476.950742:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018476.950746:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332984448:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018476.950755:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332984576:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018476.950773:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018476.950780:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018476/real 1734018476] req@ffff8800a705c380 x1818247332984576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018476.950791:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332984576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018476.950799:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018476.950803:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018476.950806:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332984576:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018476.950813:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332984704:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018476.950828:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018476.950834:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018476/real 1734018476] req@ffff8800a705f800 x1818247332984704/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018476.950844:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332984704/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018476.950852:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018476.950855:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018476.950858:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332984704:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018476.950924:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332984832:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018476.950942:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018476.950949:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018476/real 1734018476] req@ffff8800a705d880 x1818247332984832/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018476.950960:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332984832/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018476.950968:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018476.950971:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018476.950974:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332984832:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018476.950980:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332984960:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018476.950996:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018476.951002:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018476/real 1734018476] req@ffff8800a705ca80 x1818247332984960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018476.951013:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332984960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018476.951021:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018476.951023:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018476.951026:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332984960:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018476.951032:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332985088:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018476.951044:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018476.951050:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018476/real 1734018476] req@ffff8800a705df80 x1818247332985088/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018476.951060:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332985088/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018476.951067:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018476.951069:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018476.951072:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332985088:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018476.951078:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332985216:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018476.951090:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018476.951097:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018476/real 1734018476] req@ffff8800a705c700 x1818247332985216/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018476.951113:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332985216/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018476.951121:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018476.951123:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018476.951126:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332985216:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018477.660460:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018478.662474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018479.664450:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018480.666488:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018481.668481:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018481.958350:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018481.958359:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018481.958363:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018481.958373:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2744 00000100:00080000:0.0:1734018481.958383:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018481.958390:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018481.958426:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332985344/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018481.958461:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018481.958465:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018481.958467:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018481.958472:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2744 00000100:00080000:0.0:1734018481.958475:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018481.958479:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018481.958489:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018481.958500:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332985472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018481.958510:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018481.958514:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018481.958515:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018481.958519:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2744 00000100:00080000:0.0:1734018481.958522:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018481.958525:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018481.958530:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018481.958539:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332985600/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018481.958548:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018481.958551:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018481.958553:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018481.958556:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2744 00000100:00080000:0.0:1734018481.958558:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018481.958562:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018481.958566:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018481.958567:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332985344:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018481.958574:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332985728/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018481.958583:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018481.958586:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018481.958588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018481.958591:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2744 00000100:00080000:0.0:1734018481.958595:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018481.958598:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018481.958603:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018481.958610:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332985856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018481.958618:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018481.958621:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018481.958623:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018481.958627:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2744 00000100:00080000:0.0:1734018481.958629:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018481.958633:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018481.958636:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734018481.958637:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018481.958644:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332985984/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018481.958652:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018481.958655:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018481.958656:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018481.958658:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018481/real 1734018481] req@ffff8800a705d500 x1818247332985344/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018481.958660:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2744 00000100:00080000:0.0:1734018481.958662:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018481.958665:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018481.958670:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018481.958677:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332985344/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018481.958678:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332986112/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018481.958688:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018481.958691:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018481.958696:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332985344:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018481.958704:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332985472:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018481.958721:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018481.958728:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018481/real 1734018481] req@ffff8800a705e680 x1818247332985472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018481.958739:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332985472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018481.958747:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018481.958750:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018481.958754:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332985472:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018481.958759:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332985600:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018481.958773:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018481.958779:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018481/real 1734018481] req@ffff8800a705c700 x1818247332985600/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018481.958833:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332985600/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018481.958842:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018481.958844:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018481.958848:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332985600:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018481.958873:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332985728:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018481.958889:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018481.958896:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018481/real 1734018481] req@ffff8800a705df80 x1818247332985728/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018481.958907:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332985728/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018481.958915:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018481.958917:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018481.958920:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332985728:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018481.958925:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332985856:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018481.958940:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018481.958946:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018481/real 1734018481] req@ffff8800a705ca80 x1818247332985856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018481.958957:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332985856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018481.958964:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018481.958967:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018481.958970:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332985856:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018481.958975:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332985984:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018481.958986:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018481.958992:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018481/real 1734018481] req@ffff8800a705d880 x1818247332985984/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018481.959001:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332985984/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018481.959008:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018481.959011:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018481.959014:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332985984:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018481.959019:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332986112:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018481.959031:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018481.959037:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018481/real 1734018481] req@ffff8800a705f800 x1818247332986112/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018481.959053:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332986112/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018481.959061:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018481.959063:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018481.959066:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332986112:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018482.670515:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018483.672472:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018484.674502:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018485.676492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018486.678506:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018486.966333:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018486.966344:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018486.966349:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018486.966362:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2749 00000100:00080000:0.0:1734018486.966374:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018486.966384:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018486.966433:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332986240/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018486.966472:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018486.966476:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018486.966478:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018486.966484:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2749 00000100:00080000:0.0:1734018486.966487:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018486.966492:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018486.966503:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018486.966515:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332986368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018486.966526:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018486.966529:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018486.966531:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018486.966536:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2749 00000100:00080000:0.0:1734018486.966539:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018486.966542:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018486.966549:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018486.966558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332986496/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018486.966567:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018486.966570:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018486.966572:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018486.966576:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2749 00000100:00080000:0.0:1734018486.966578:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018486.966582:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018486.966587:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018486.966596:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332986624/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018486.966605:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018486.966609:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018486.966610:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018486.966611:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332986240:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018486.966614:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2749 00000100:00080000:0.0:1734018486.966617:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018486.966621:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018486.966625:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018486.966634:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332986752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018486.966643:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018486.966646:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018486.966648:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018486.966653:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2749 00000100:00080000:0.0:1734018486.966655:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018486.966659:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018486.966663:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018486.966671:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332986880/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018486.966679:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018486.966682:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018486.966684:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018486.966687:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2749 00000100:00080000:0.0:1734018486.966690:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018486.966694:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018486.966695:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734018486.966698:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018486.966746:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332987008/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018486.966767:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018486/real 1734018486] req@ffff8800a705c380 x1818247332986240/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018486.966787:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332986240/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018486.966799:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018486.966803:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018486.966808:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332986240:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018486.966816:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332986368:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018486.966838:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018486.966845:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018486/real 1734018486] req@ffff8800a705d180 x1818247332986368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018486.966856:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332986368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018486.966865:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018486.966868:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018486.966872:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332986368:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018486.966879:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332986496:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018486.966893:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018486.966900:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018486/real 1734018486] req@ffff8800a705dc00 x1818247332986496/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018486.966910:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332986496/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018486.966918:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018486.966920:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018486.966924:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332986496:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018486.966951:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332986624:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018486.966964:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018486.966970:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018486/real 1734018486] req@ffff8800a705f100 x1818247332986624/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018486.966980:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332986624/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018486.966988:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018486.966991:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018486.966994:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332986624:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018486.967000:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332986752:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018486.967016:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018486.967022:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018486/real 1734018486] req@ffff8800a705fb80 x1818247332986752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018486.967032:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332986752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018486.967040:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018486.967043:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018486.967046:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332986752:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018486.967052:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332986880:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018486.967064:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018486.967070:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018486/real 1734018486] req@ffff8800a705ea00 x1818247332986880/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018486.967080:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332986880/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018486.967087:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018486.967090:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018486.967092:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332986880:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018486.967098:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332987008:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018486.967110:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018486.967116:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018486/real 1734018486] req@ffff8800a705ed80 x1818247332987008/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018486.967132:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332987008/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018486.967140:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018486.967142:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018486.967145:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332987008:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018487.680457:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018488.682505:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018489.684454:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018490.686508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018491.688515:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018491.974326:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018491.974336:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018491.974340:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018491.974351:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2754 00000100:00080000:0.0:1734018491.974361:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018491.974369:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018491.974405:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332987136/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018491.974447:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018491.974452:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018491.974454:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018491.974458:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2754 00000100:00080000:0.0:1734018491.974462:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018491.974466:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018491.974476:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018491.974487:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332987264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018491.974497:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018491.974500:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018491.974501:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018491.974506:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2754 00000100:00080000:0.0:1734018491.974508:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018491.974511:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018491.974516:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018491.974526:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332987392/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018491.974534:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018491.974537:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018491.974538:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018491.974542:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2754 00000100:00080000:0.0:1734018491.974544:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018491.974548:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018491.974552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018491.974561:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332987520/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018491.974569:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018491.974571:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332987136:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018491.974573:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018491.974574:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018491.974578:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2754 00000100:00080000:0.0:1734018491.974580:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018491.974583:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018491.974587:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018491.974595:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332987648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018491.974603:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018491.974606:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018491.974608:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018491.974611:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2754 00000100:00080000:0.0:1734018491.974668:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018491.974673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018491.974678:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018491.974684:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018491.974687:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332987776/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018491.974696:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018491.974700:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018491.974702:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018491.974706:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2754 00000100:00100000:1.0:1734018491.974708:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018491/real 1734018491] req@ffff8800a705e300 x1818247332987136/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018491.974709:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018491.974712:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018491.974716:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018491.974724:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332987904/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018491.974727:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332987136/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018491.974738:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018491.974741:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018491.974746:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332987136:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018491.974753:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332987264:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018491.974772:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018491.974779:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018491/real 1734018491] req@ffff8800a705c000 x1818247332987264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018491.974790:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332987264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018491.974798:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018491.974801:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018491.974805:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332987264:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018491.974811:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332987392:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018491.974825:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018491.974831:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018491/real 1734018491] req@ffff8800a705ed80 x1818247332987392/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018491.974841:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332987392/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018491.974849:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018491.974851:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018491.974854:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332987392:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018491.974878:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332987520:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018491.974890:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018491.974896:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018491/real 1734018491] req@ffff8800a705ea00 x1818247332987520/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018491.974907:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332987520/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018491.974914:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018491.974916:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018491.974919:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332987520:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018491.974925:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332987648:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018491.974940:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018491.974946:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018491/real 1734018491] req@ffff8800a705fb80 x1818247332987648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018491.974956:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332987648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018491.974963:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018491.974965:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018491.974968:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332987648:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018491.974974:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332987776:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018491.974985:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018491.974991:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018491/real 1734018491] req@ffff8800a705f100 x1818247332987776/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018491.975001:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332987776/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018491.975008:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018491.975011:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018491.975013:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332987776:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018491.975018:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332987904:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018491.975030:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018491.975036:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018491/real 1734018491] req@ffff8800a705dc00 x1818247332987904/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018491.975052:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332987904/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018491.975059:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018491.975061:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018491.975064:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332987904:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018492.690491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018493.692473:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018494.694443:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018495.696515:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018496.698526:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018496.982327:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018496.982338:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018496.982342:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018496.982352:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2759 00000100:00080000:0.0:1734018496.982362:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018496.982371:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018496.982406:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332988032/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018496.982439:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018496.982444:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018496.982446:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018496.982450:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2759 00000100:00080000:0.0:1734018496.982453:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018496.982457:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018496.982467:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018496.982478:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332988160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018496.982490:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018496.982493:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018496.982495:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018496.982498:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2759 00000100:00080000:0.0:1734018496.982501:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018496.982504:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018496.982509:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018496.982517:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332988288/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018496.982525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018496.982528:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018496.982530:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018496.982568:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2759 00000100:00080000:0.0:1734018496.982571:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018496.982574:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018496.982579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018496.982584:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332988032:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018496.982589:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332988416/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018496.982598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018496.982601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018496.982603:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018496.982606:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2759 00000100:00080000:0.0:1734018496.982610:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018496.982613:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018496.982618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018496.982626:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332988544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018496.982634:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018496.982637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018496.982638:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018496.982642:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2759 00000100:00080000:0.0:1734018496.982645:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018496.982648:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018496.982652:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018496.982652:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018496.982660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332988672/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018496.982668:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018496.982671:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018496.982673:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018496.982674:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018496/real 1734018496] req@ffff8800a705d180 x1818247332988032/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018496.982676:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2759 00000100:00080000:0.0:1734018496.982679:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018496.982682:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018496.982686:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018496.982694:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332988032/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018496.982695:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332988800/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018496.982705:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018496.982708:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018496.982713:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332988032:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018496.982721:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332988160:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018496.982738:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018496.982744:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018496/real 1734018496] req@ffff8800a705c380 x1818247332988160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018496.982755:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332988160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018496.982763:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018496.982767:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018496.982770:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332988160:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018496.982776:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332988288:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018496.982790:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018496.982796:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018496/real 1734018496] req@ffff8800a705f800 x1818247332988288/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018496.982806:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332988288/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018496.982813:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018496.982816:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018496.982819:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332988288:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018496.982844:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332988416:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018496.982856:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018496.982862:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018496/real 1734018496] req@ffff8800a705d880 x1818247332988416/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018496.982872:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332988416/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018496.982880:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018496.982883:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018496.982885:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332988416:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018496.982891:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332988544:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018496.982906:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018496.982912:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018496/real 1734018496] req@ffff8800a705ca80 x1818247332988544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018496.982923:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332988544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018496.982931:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018496.982933:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018496.982936:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332988544:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018496.982941:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332988672:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018496.982953:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018496.982959:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018496/real 1734018496] req@ffff8800a705df80 x1818247332988672/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018496.982969:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332988672/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018496.982976:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018496.982978:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018496.982980:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332988672:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018496.982986:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332988800:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018496.982997:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018496.983003:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018496/real 1734018496] req@ffff8800a705c700 x1818247332988800/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018496.983020:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332988800/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018496.983027:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018496.983030:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018496.983033:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332988800:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018497.700457:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018498.702467:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018499.704549:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018500.706535:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018501.708406:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018501.990361:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018501.990370:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018501.990375:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018501.990385:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2764 00000100:00080000:0.0:1734018501.990396:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018501.990404:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018501.990439:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332988928/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018501.990505:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018501.990509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018501.990511:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018501.990516:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2764 00000100:00080000:0.0:1734018501.990519:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018501.990523:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018501.990533:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018501.990545:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332989056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018501.990556:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018501.990559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018501.990561:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018501.990564:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2764 00000100:00080000:0.0:1734018501.990567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018501.990570:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018501.990576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018501.990586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332989184/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018501.990594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018501.990597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018501.990599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018501.990602:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2764 00000100:00080000:0.0:1734018501.990604:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018501.990608:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018501.990612:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018501.990620:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332989312/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018501.990621:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332988928:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018501.990629:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018501.990633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018501.990635:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018501.990638:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2764 00000100:00080000:0.0:1734018501.990641:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018501.990644:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018501.990649:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018501.990656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332989440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018501.990665:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018501.990668:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018501.990670:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018501.990674:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2764 00000100:00080000:0.0:1734018501.990677:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018501.990680:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018501.990685:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018501.990691:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018501.990692:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332989568/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018501.990700:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018501.990703:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018501.990705:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018501.990708:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2764 00000100:00080000:0.0:1734018501.990711:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018501.990714:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018501.990714:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018501/real 1734018501] req@ffff8800a705e680 x1818247332988928/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018501.990718:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018501.990726:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332989696/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018501.990733:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332988928/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018501.990744:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018501.990747:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018501.990752:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332988928:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018501.990759:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332989056:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018501.990776:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018501.990783:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018501/real 1734018501] req@ffff8800a705d500 x1818247332989056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018501.990794:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332989056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018501.990802:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018501.990805:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018501.990809:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332989056:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018501.990815:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332989184:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018501.990829:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018501.990834:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018501/real 1734018501] req@ffff8800a705c700 x1818247332989184/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018501.990845:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332989184/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018501.990853:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018501.990855:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018501.990858:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332989184:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018501.990883:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332989312:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018501.990895:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018501.990901:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018501/real 1734018501] req@ffff8800a705df80 x1818247332989312/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018501.990911:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332989312/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018501.990919:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018501.990921:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018501.990924:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332989312:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018501.990929:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332989440:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018501.990944:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018501.990950:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018501/real 1734018501] req@ffff8800a705ca80 x1818247332989440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018501.990961:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332989440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018501.990969:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018501.990971:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018501.990974:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332989440:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018501.990979:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332989568:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018501.990991:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018501.990997:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018501/real 1734018501] req@ffff8800a705d880 x1818247332989568/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018501.991006:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332989568/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018501.991013:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018501.991015:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018501.991018:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332989568:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018501.991023:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332989696:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018501.991035:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018501.991041:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018501/real 1734018501] req@ffff8800a705f800 x1818247332989696/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018501.991057:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332989696/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018501.991065:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018501.991067:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018501.991070:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332989696:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018502.710490:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018503.712446:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018504.714494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018505.716472:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018506.718505:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018506.998334:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018506.998345:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018506.998350:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018506.998405:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2769 00000100:00080000:0.0:1734018506.998418:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018506.998427:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018506.998475:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332989824/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018506.998521:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018506.998526:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018506.998528:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018506.998532:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2769 00000100:00080000:0.0:1734018506.998536:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018506.998540:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018506.998552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018506.998564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332989952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018506.998574:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018506.998578:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018506.998579:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018506.998583:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2769 00000100:00080000:0.0:1734018506.998586:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018506.998590:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018506.998595:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018506.998604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332990080/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018506.998612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018506.998616:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018506.998617:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018506.998621:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2769 00000100:00080000:0.0:1734018506.998623:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018506.998628:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018506.998632:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018506.998641:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332990208/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018506.998651:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018506.998654:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018506.998655:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018506.998656:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332989824:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018506.998660:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2769 00000100:00080000:0.0:1734018506.998662:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018506.998666:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018506.998670:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018506.998679:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332990336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018506.998687:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018506.998691:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018506.998692:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018506.998696:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2769 00000100:00080000:0.0:1734018506.998698:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018506.998702:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018506.998706:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018506.998714:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332990464/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018506.998723:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018506.998727:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018506.998729:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734018506.998732:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018506.998733:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2769 00000100:00080000:0.0:1734018506.998736:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018506.998740:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018506.998744:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018506.998754:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332990592/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018506.998756:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018506/real 1734018506] req@ffff8800a705c380 x1818247332989824/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018506.998776:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332989824/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018506.998789:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018506.998793:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018506.998798:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332989824:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018506.998806:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332989952:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018506.998824:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018506.998831:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018506/real 1734018506] req@ffff8800a705d180 x1818247332989952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018506.998842:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332989952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018506.998850:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018506.998854:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018506.998858:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332989952:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018506.998865:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332990080:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018506.998879:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018506.998885:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018506/real 1734018506] req@ffff8800a705dc00 x1818247332990080/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018506.998895:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332990080/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018506.998903:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018506.998905:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018506.998909:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332990080:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018506.998935:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332990208:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018506.998948:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018506.998954:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018506/real 1734018506] req@ffff8800a705f100 x1818247332990208/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018506.998965:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332990208/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018506.998973:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018506.998975:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018506.998978:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332990208:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018506.998984:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332990336:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018506.999000:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018506.999006:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018506/real 1734018506] req@ffff8800a705fb80 x1818247332990336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018506.999016:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332990336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018506.999023:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018506.999026:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018506.999029:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332990336:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018506.999035:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332990464:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018506.999047:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018506.999053:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018506/real 1734018506] req@ffff8800a705ea00 x1818247332990464/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018506.999063:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332990464/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018506.999070:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018506.999072:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018506.999075:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332990464:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018506.999081:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332990592:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018506.999094:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018506.999100:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018506/real 1734018506] req@ffff8800a705ed80 x1818247332990592/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018506.999116:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332990592/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018506.999124:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018506.999126:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018506.999129:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332990592:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018507.720488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018508.722519:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018509.724421:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018510.726546:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018511.728498:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018512.006359:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018512.006369:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018512.006373:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018512.006384:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2774 00000100:00080000:0.0:1734018512.006396:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018512.006404:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018512.006440:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332990720/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018512.006474:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018512.006478:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018512.006480:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018512.006485:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2774 00000100:00080000:0.0:1734018512.006488:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018512.006492:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018512.006501:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018512.006512:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332990848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018512.006523:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018512.006526:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018512.006528:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018512.006531:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2774 00000100:00080000:0.0:1734018512.006534:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018512.006537:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018512.006542:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018512.006551:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332990976/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018512.006560:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018512.006563:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018512.006564:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018512.006568:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2774 00000100:00080000:0.0:1734018512.006570:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018512.006573:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018512.006578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018512.006586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332991104/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018512.006595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018512.006597:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332990720:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018512.006598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018512.006599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018512.006603:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2774 00000100:00080000:0.0:1734018512.006606:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018512.006609:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018512.006614:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018512.006621:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332991232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018512.006630:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018512.006632:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018512.006634:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018512.006638:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2774 00000100:00080000:0.0:1734018512.006652:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018512.006656:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018512.006660:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018512.006668:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332991360/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018512.006676:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018512.006679:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018512.006681:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018512.006684:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2774 00000100:00080000:0.0:1734018512.006687:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018512.006690:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018512.006694:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018512.006696:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332990848:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018512.006702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332991488/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018512.006718:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332990976:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018512.006741:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332991104:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018512.006756:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332991232:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018512.006770:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332991360:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018512.006784:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332991488:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018512.730520:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018513.732507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018514.734487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1734018515.012554:0:1955:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:2.0:1734018515.012610:0:1955:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1734018515.012636:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018515.012681:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018515.012690:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018515.012699:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018515.012706:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018515.012713:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018515.012720:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734018515.012799:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018512/real 1734018515] req@ffff8800a705ea00 x1818247332991104/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018515.012831:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332991104/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018515.012845:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018515.012851:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018515.012859:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332991104:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018515.012872:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018512/real 1734018515] req@ffff8800a705fb80 x1818247332991232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018515.012891:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332991232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018515.012904:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018515.012908:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018515.012913:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332991232:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018515.012925:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018512/real 1734018515] req@ffff8800a705f100 x1818247332991360/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018515.012941:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332991360/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018515.012954:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018515.012958:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018515.012964:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332991360:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018515.012975:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018512/real 1734018515] req@ffff8800a705dc00 x1818247332991488/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018515.012993:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332991488/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018515.013005:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018515.013009:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018515.013014:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332991488:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018515.013030:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018512/real 1734018515] req@ffff8800a705c000 x1818247332990720/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018515.013050:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332990720/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018515.013063:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018515.013066:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018515.013094:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332990720:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018515.013106:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018512/real 1734018515] req@ffff8800a705e300 x1818247332990848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018515.013127:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332990848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018515.013140:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018515.013151:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018515.013157:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332990848:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018515.013166:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018512/real 1734018515] req@ffff8800a705ed80 x1818247332990976/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018515.013184:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332990976/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018515.013195:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018515.013199:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018515.013204:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332990976:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018515.736501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018516.738514:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018517.014446:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018517.014457:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018517.014463:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018517.014476:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2779 00000100:00080000:2.0:1734018517.014488:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018517.014497:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018517.014545:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247332991616/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018517.014591:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018517.014596:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018517.014598:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018517.014603:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2779 00000100:00080000:2.0:1734018517.014606:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018517.014611:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018517.014623:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018517.014637:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247332991744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018517.014648:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018517.014651:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018517.014653:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018517.014659:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2779 00000100:00080000:2.0:1734018517.014662:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018517.014665:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018517.014670:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018517.014679:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247332991872/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018517.014687:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018517.014691:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018517.014692:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018517.014696:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2779 00000100:00080000:2.0:1734018517.014698:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018517.014702:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018517.014707:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018517.014716:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247332992000/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018517.014719:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332991616:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734018517.014726:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018517.014730:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018517.014731:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018517.014735:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2779 00000100:00080000:2.0:1734018517.014738:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018517.014741:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018517.014746:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018517.014755:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247332992128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018517.014765:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018517.014768:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018517.014770:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018517.014774:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2779 00000100:00080000:2.0:1734018517.014777:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018517.014781:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018517.014785:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018517.014793:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247332992256/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018517.014797:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:2.0:1734018517.014801:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018517.014804:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018517.014805:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018517.014809:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2779 00000100:00080000:2.0:1734018517.014812:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018517.014816:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018517.014820:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018517.014823:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018517/real 1734018517] req@ffff8800a73ca680 x1818247332991616/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018517.014828:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247332992384/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018517.014844:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ca680 x1818247332991616/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018517.014856:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018517.014859:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018517.014864:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332991616:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018517.014872:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332991744:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018517.014891:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018517.014898:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018517/real 1734018517] req@ffff8800a73c9f80 x1818247332991744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018517.014909:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9f80 x1818247332991744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018517.014917:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018517.014920:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018517.014923:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332991744:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018517.014930:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332991872:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018517.014944:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018517.014949:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018517/real 1734018517] req@ffff8800a73caa00 x1818247332991872/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018517.014959:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73caa00 x1818247332991872/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018517.014968:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018517.014971:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018517.014974:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332991872:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018517.015000:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332992000:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018517.015013:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018517.015019:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018517/real 1734018517] req@ffff8800a73c8e00 x1818247332992000/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018517.015029:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247332992000/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018517.015037:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018517.015040:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018517.015043:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332992000:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018517.015048:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332992128:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018517.015061:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018517.015066:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018517/real 1734018517] req@ffff8800a73c8000 x1818247332992128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018517.015076:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247332992128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018517.015084:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018517.015086:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018517.015089:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332992128:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018517.015096:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332992256:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018517.015109:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018517.015115:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018517/real 1734018517] req@ffff8800a73c9500 x1818247332992256/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018517.015125:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247332992256/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018517.015132:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018517.015134:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018517.015137:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332992256:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018517.015143:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332992384:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018517.015155:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018517.015161:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018517/real 1734018517] req@ffff8800a73c9c00 x1818247332992384/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018517.015170:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247332992384/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018517.015177:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018517.015180:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018517.015183:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332992384:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018517.740465:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018518.742481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018519.744468:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018520.746500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018521.748489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018522.022522:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018522.022532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018522.022537:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018522.022547:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2784 00000100:00080000:0.0:1734018522.022557:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018522.022565:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018522.022600:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332992512/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018522.022637:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018522.022641:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018522.022644:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018522.022648:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2784 00000100:00080000:0.0:1734018522.022651:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018522.022655:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018522.022665:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018522.022676:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332992640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018522.022686:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018522.022689:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018522.022691:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018522.022695:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2784 00000100:00080000:0.0:1734018522.022697:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018522.022701:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018522.022706:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018522.022714:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332992768/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018522.022723:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018522.022726:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018522.022728:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018522.022731:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2784 00000100:00080000:0.0:1734018522.022733:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018522.022736:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018522.022741:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018522.022749:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332992896/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018522.022752:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332992512:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018522.022757:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018522.022761:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018522.022762:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018522.022766:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2784 00000100:00080000:0.0:1734018522.022768:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018522.022771:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018522.022776:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018522.022783:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332993024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018522.022792:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018522.022795:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018522.022797:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018522.022801:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2784 00000100:00080000:0.0:1734018522.022806:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018522.022809:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018522.022813:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018522.022821:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018522.022822:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332993152/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018522.022830:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018522.022833:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018522.022835:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018522.022838:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2784 00000100:00080000:0.0:1734018522.022841:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018522.022842:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018522/real 1734018522] req@ffff8800a705d180 x1818247332992512/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018522.022844:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018522.022849:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018522.022856:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332993280/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018522.022862:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332992512/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018522.022873:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018522.022876:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018522.022881:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332992512:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018522.022888:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332992640:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018522.022905:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018522.022912:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018522/real 1734018522] req@ffff8800a705c380 x1818247332992640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018522.022923:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332992640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018522.022931:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018522.022933:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018522.022937:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332992640:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018522.022942:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332992768:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018522.022956:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018522.022962:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018522/real 1734018522] req@ffff8800a705f800 x1818247332992768/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018522.022972:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332992768/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018522.022979:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018522.022982:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018522.022985:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332992768:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018522.023010:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332992896:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018522.023023:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018522.023029:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018522/real 1734018522] req@ffff8800a705d880 x1818247332992896/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018522.023039:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332992896/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018522.023047:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018522.023049:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018522.023052:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332992896:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018522.023057:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332993024:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018522.023069:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018522.023075:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018522/real 1734018522] req@ffff8800a705ca80 x1818247332993024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018522.023084:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332993024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018522.023092:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018522.023094:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018522.023097:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332993024:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018522.023103:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332993152:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018522.023174:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018522.023182:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018522/real 1734018522] req@ffff8800a705df80 x1818247332993152/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018522.023193:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332993152/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018522.023201:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018522.023203:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018522.023206:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332993152:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018522.023211:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332993280:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018522.023276:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018522.023284:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018522/real 1734018522] req@ffff8800a705c700 x1818247332993280/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018522.023295:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332993280/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018522.023302:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018522.023304:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018522.023307:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332993280:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018522.750513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018523.752480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018524.754481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018525.756460:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018526.758486:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018527.030338:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018527.030351:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018527.030357:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018527.030371:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2789 00000100:00080000:0.0:1734018527.030382:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018527.030392:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018527.030442:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332993408/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018527.030489:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018527.030494:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018527.030496:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018527.030501:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2789 00000100:00080000:0.0:1734018527.030505:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018527.030509:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018527.030521:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018527.030534:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332993536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018527.030544:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018527.030547:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018527.030549:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018527.030553:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2789 00000100:00080000:0.0:1734018527.030555:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018527.030559:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018527.030564:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018527.030575:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332993664/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018527.030584:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018527.030587:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018527.030588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018527.030592:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2789 00000100:00080000:0.0:1734018527.030594:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018527.030599:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018527.030603:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018527.030613:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332993792/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018527.030616:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332993408:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018527.030623:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018527.030626:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018527.030628:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018527.030632:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2789 00000100:00080000:0.0:1734018527.030634:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018527.030638:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018527.030642:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018527.030651:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332993920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018527.030659:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018527.030662:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018527.030664:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018527.030668:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2789 00000100:00080000:0.0:1734018527.030681:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018527.030685:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018527.030690:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018527.030694:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018527.030699:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332994048/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018527.030707:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018527.030711:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018527.030713:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018527.030717:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2789 00000100:00100000:1.0:1734018527.030720:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018527/real 1734018527] req@ffff8800a705d500 x1818247332993408/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018527.030721:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018527.030724:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018527.030729:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018527.030738:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332994176/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018527.030747:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332993408/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018527.030758:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018527.030762:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018527.030767:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332993408:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018527.030775:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332993536:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018527.030793:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018527.030800:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018527/real 1734018527] req@ffff8800a705e680 x1818247332993536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018527.030811:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332993536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018527.030819:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018527.030822:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018527.030825:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332993536:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018527.030832:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332993664:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018527.030846:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018527.030852:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018527/real 1734018527] req@ffff8800a705c700 x1818247332993664/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018527.030862:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332993664/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018527.030869:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018527.030873:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018527.030876:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332993664:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018527.030902:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332993792:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018527.030916:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018527.030922:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018527/real 1734018527] req@ffff8800a705df80 x1818247332993792/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018527.030933:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332993792/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018527.030940:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018527.030942:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018527.030945:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332993792:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018527.030951:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332993920:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018527.030963:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018527.030969:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018527/real 1734018527] req@ffff8800a705ca80 x1818247332993920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018527.030979:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332993920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018527.030987:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018527.030990:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018527.030993:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332993920:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018527.030999:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332994048:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018527.031015:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018527.031069:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018527/real 1734018527] req@ffff8800a705d880 x1818247332994048/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018527.031083:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332994048/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018527.031091:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018527.031093:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018527.031097:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332994048:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018527.031102:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332994176:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018527.031119:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018527.031126:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018527/real 1734018527] req@ffff8800a705f800 x1818247332994176/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018527.031136:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332994176/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018527.031144:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018527.031147:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018527.031149:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332994176:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018527.760496:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018528.762478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018529.764500:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018530.766500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018531.768516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018532.038439:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018532.038449:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018532.038453:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018532.038464:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2794 00000100:00080000:2.0:1734018532.038474:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018532.038482:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018532.038517:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247332994304/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018532.038555:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018532.038559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018532.038561:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018532.038566:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2794 00000100:00080000:2.0:1734018532.038569:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018532.038573:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018532.038582:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018532.038593:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247332994432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018532.038603:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018532.038607:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018532.038608:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018532.038612:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2794 00000100:00080000:2.0:1734018532.038615:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018532.038618:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018532.038624:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018532.038632:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247332994560/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018532.038641:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018532.038644:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018532.038645:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018532.038648:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2794 00000100:00080000:2.0:1734018532.038651:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018532.038654:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018532.038659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018532.038665:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332994304:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734018532.038668:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247332994688/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018532.038676:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018532.038679:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018532.038681:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018532.038684:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2794 00000100:00080000:2.0:1734018532.038687:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018532.038690:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018532.038695:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018532.038702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247332994816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018532.038711:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018532.038714:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018532.038731:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734018532.038734:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:2.0:1734018532.038736:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2794 00000100:00080000:2.0:1734018532.038739:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018532.038743:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018532.038748:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018532.038756:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018532/real 1734018532] req@ffff8800a73cad80 x1818247332994304/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018532.038756:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247332994944/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018532.038765:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018532.038768:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018532.038770:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018532.038773:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2794 00000100:00080000:2.0:1734018532.038776:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018532.038779:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018532.038783:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018532.038784:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247332994304/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018532.038790:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247332995072/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018532.038794:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018532.038797:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018532.038802:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332994304:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018532.038809:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332994432:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018532.038826:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018532.038833:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018532/real 1734018532] req@ffff8800a73cb100 x1818247332994432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018532.038844:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247332994432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018532.038852:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018532.038854:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018532.038858:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332994432:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018532.038863:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332994560:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018532.038876:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018532.038882:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018532/real 1734018532] req@ffff8800a73c9880 x1818247332994560/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018532.038892:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247332994560/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018532.038899:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018532.038902:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018532.038905:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332994560:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018532.038930:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332994688:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018532.038988:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018532.038996:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018532/real 1734018532] req@ffff8800a73c9c00 x1818247332994688/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018532.039007:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247332994688/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018532.039015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018532.039017:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018532.039020:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332994688:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018532.039026:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332994816:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018532.039038:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018532.039045:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018532/real 1734018532] req@ffff8800a73c9500 x1818247332994816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018532.039055:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247332994816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018532.039063:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018532.039065:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018532.039068:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332994816:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018532.039073:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332994944:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018532.039088:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018532.039094:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018532/real 1734018532] req@ffff8800a73c8000 x1818247332994944/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018532.039105:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247332994944/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018532.039112:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018532.039114:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018532.039117:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332994944:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018532.039123:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332995072:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018532.039134:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018532.039140:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018532/real 1734018532] req@ffff8800a73c8e00 x1818247332995072/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018532.039150:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247332995072/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018532.039157:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018532.039159:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018532.039162:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332995072:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018532.770484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018533.772438:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018534.774481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018535.776506:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018536.778495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018537.046364:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018537.046372:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018537.046377:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018537.046388:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2799 00000100:00080000:2.0:1734018537.046398:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018537.046405:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018537.046442:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247332995200/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018537.046476:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018537.046480:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018537.046482:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018537.046486:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2799 00000100:00080000:2.0:1734018537.046490:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018537.046494:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018537.046503:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018537.046514:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247332995328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018537.046524:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018537.046527:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018537.046529:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018537.046532:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2799 00000100:00080000:2.0:1734018537.046535:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018537.046539:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018537.046544:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018537.046553:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247332995456/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018537.046562:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018537.046565:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018537.046567:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018537.046571:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2799 00000100:00080000:2.0:1734018537.046573:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018537.046576:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018537.046581:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018537.046584:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332995200:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734018537.046589:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247332995584/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018537.046598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018537.046601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018537.046603:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018537.046606:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2799 00000100:00080000:2.0:1734018537.046608:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018537.046612:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018537.046616:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018537.046624:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247332995712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018537.046632:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018537.046635:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018537.046637:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018537.046641:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2799 00000100:00080000:2.0:1734018537.046644:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018537.046647:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018537.046651:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018537.046655:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018537.046659:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247332995840/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018537.046668:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018537.046671:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018537.046673:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018537.046676:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2799 00000100:00100000:1.0:1734018537.046677:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018537/real 1734018537] req@ffff8800a73caa00 x1818247332995200/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018537.046679:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018537.046682:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018537.046686:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018537.046693:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247332995968/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018537.046703:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73caa00 x1818247332995200/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018537.046714:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018537.046717:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018537.046721:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332995200:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018537.046729:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332995328:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018537.046746:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018537.046753:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018537/real 1734018537] req@ffff8800a73c9f80 x1818247332995328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018537.046763:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9f80 x1818247332995328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018537.046771:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018537.046774:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018537.046777:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332995328:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018537.046782:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332995456:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018537.046796:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018537.046802:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018537/real 1734018537] req@ffff8800a73ca680 x1818247332995456/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018537.046812:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ca680 x1818247332995456/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018537.046819:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018537.046822:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018537.046825:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332995456:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018537.046903:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332995584:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018537.046923:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018537.046931:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018537/real 1734018537] req@ffff8800a73c8e00 x1818247332995584/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018537.046942:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247332995584/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018537.046950:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018537.046952:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018537.046955:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332995584:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018537.046961:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332995712:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018537.046973:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018537.046979:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018537/real 1734018537] req@ffff8800a73c8000 x1818247332995712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018537.046989:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247332995712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018537.046997:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018537.047000:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018537.047003:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332995712:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018537.047008:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332995840:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018537.047023:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018537.047029:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018537/real 1734018537] req@ffff8800a73c9500 x1818247332995840/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018537.047039:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247332995840/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018537.047047:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018537.047049:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018537.047052:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332995840:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018537.047057:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332995968:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018537.047068:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018537.047074:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018537/real 1734018537] req@ffff8800a73c9c00 x1818247332995968/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018537.047084:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247332995968/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018537.047091:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018537.047093:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018537.047096:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332995968:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018537.780471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018538.782474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018539.784510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018540.786381:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018541.788478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018542.054526:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018542.054537:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018542.054542:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018542.054556:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2804 00000100:00080000:0.0:1734018542.054568:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018542.054578:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018542.054627:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332996096/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018542.054671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018542.054676:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018542.054678:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018542.054683:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2804 00000100:00080000:0.0:1734018542.054687:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018542.054691:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018542.054703:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018542.054715:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332996224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018542.054727:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018542.054730:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018542.054732:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018542.054736:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2804 00000100:00080000:0.0:1734018542.054739:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018542.054742:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018542.054747:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018542.054756:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332996352/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018542.054807:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018542.054811:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018542.054813:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018542.054817:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2804 00000100:00080000:0.0:1734018542.054820:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018542.054824:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018542.054830:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018542.054838:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332996096:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018542.054840:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332996480/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018542.054851:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018542.054855:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018542.054857:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018542.054861:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2804 00000100:00080000:0.0:1734018542.054865:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018542.054869:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018542.054874:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018542.054883:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332996608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018542.054892:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018542.054895:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018542.054897:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018542.054901:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2804 00000100:00080000:0.0:1734018542.054905:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018542.054908:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018542.054913:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018542.054917:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018542.054921:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332996736/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018542.054929:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018542.054932:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018542.054934:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018542.054938:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2804 00000100:00080000:0.0:1734018542.054940:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018542.054943:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018542/real 1734018542] req@ffff8800a705c380 x1818247332996096/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018542.054944:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018542.054948:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018542.054958:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332996864/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018542.054971:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332996096/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018542.054982:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018542.054985:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018542.054990:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332996096:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018542.054998:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332996224:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018542.055017:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018542.055024:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018542/real 1734018542] req@ffff8800a705d180 x1818247332996224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018542.055034:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332996224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018542.055043:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018542.055045:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018542.055049:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332996224:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018542.055055:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332996352:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018542.055069:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018542.055075:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018542/real 1734018542] req@ffff8800a705ed80 x1818247332996352/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018542.055085:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332996352/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018542.055093:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018542.055096:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018542.055099:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332996352:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018542.055127:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332996480:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018542.055140:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018542.055146:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018542/real 1734018542] req@ffff8800a705e300 x1818247332996480/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018542.055157:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332996480/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018542.055164:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018542.055167:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018542.055169:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332996480:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018542.055176:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332996608:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018542.055187:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018542.055193:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018542/real 1734018542] req@ffff8800a705c000 x1818247332996608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018542.055203:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332996608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018542.055211:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018542.055213:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018542.055263:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332996608:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018542.055271:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332996736:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018542.055292:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018542.055299:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018542/real 1734018542] req@ffff8800a705dc00 x1818247332996736/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018542.055310:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332996736/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018542.055318:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018542.055320:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018542.055323:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332996736:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018542.055329:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332996864:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018542.055341:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018542.055347:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018542/real 1734018542] req@ffff8800a705f100 x1818247332996864/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018542.055357:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332996864/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018542.055364:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018542.055367:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018542.055370:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332996864:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018542.790498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018543.792417:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018544.794452:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018545.796488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018546.798489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018547.062373:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018547.062384:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018547.062390:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018547.062404:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2809 00000100:00080000:0.0:1734018547.062416:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018547.062426:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018547.062468:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247332996992/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018547.062506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018547.062510:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018547.062513:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018547.062517:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2809 00000100:00080000:0.0:1734018547.062521:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018547.062526:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018547.062536:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018547.062549:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247332997120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018547.062559:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018547.062562:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018547.062564:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018547.062568:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2809 00000100:00080000:0.0:1734018547.062570:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018547.062574:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018547.062579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018547.062589:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247332997248/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018547.062598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018547.062601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018547.062603:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018547.062606:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2809 00000100:00080000:0.0:1734018547.062608:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018547.062612:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018547.062616:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018547.062619:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332996992:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018547.062624:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247332997376/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018547.062633:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018547.062636:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018547.062638:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018547.062641:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2809 00000100:00080000:0.0:1734018547.062643:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018547.062647:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018547.062651:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018547.062658:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247332997504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018547.062667:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018547.062670:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018547.062672:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018547.062676:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2809 00000100:00080000:0.0:1734018547.062679:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018547.062716:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018547.062721:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018547.062728:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018547.062730:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247332997632/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018547.062739:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018547.062742:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018547.062744:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018547.062747:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2809 00000100:00080000:0.0:1734018547.062750:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018547.062751:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018547/real 1734018547] req@ffff8800a705fb80 x1818247332996992/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018547.062754:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018547.062758:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018547.062765:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332997760/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018547.062779:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247332996992/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018547.062790:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018547.062793:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018547.062797:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332996992:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018547.062805:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332997120:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018547.062825:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018547.062831:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018547/real 1734018547] req@ffff8800a705ea00 x1818247332997120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018547.062842:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247332997120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018547.062850:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018547.062852:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018547.062856:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332997120:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018547.062861:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332997248:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018547.062874:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018547.062880:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018547/real 1734018547] req@ffff8800a705f100 x1818247332997248/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018547.062890:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247332997248/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018547.062897:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018547.062900:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018547.062903:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332997248:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018547.062928:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332997376:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018547.062941:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018547.062947:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018547/real 1734018547] req@ffff8800a705dc00 x1818247332997376/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018547.062958:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247332997376/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018547.062965:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018547.062967:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018547.062970:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332997376:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018547.062975:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332997504:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018547.062987:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018547.062993:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018547/real 1734018547] req@ffff8800a705c000 x1818247332997504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018547.063003:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247332997504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018547.063010:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018547.063012:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018547.063015:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332997504:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018547.063021:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332997632:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018547.063036:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018547.063042:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018547/real 1734018547] req@ffff8800a705e300 x1818247332997632/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018547.063052:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247332997632/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018547.063059:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018547.063061:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018547.063064:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332997632:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018547.063069:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332997760:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018547.063080:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018547.063086:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018547/real 1734018547] req@ffff8800a705ed80 x1818247332997760/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018547.063096:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332997760/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018547.063103:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018547.063105:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018547.063108:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332997760:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018547.800484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018548.802436:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018549.804510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018550.806484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018551.808464:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018552.070372:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018552.070386:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018552.070392:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018552.070406:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2814 00000100:00080000:0.0:1734018552.070419:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018552.070429:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018552.070477:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332997888/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018552.070525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018552.070530:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018552.070532:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018552.070537:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2814 00000100:00080000:0.0:1734018552.070542:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018552.070546:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018552.070558:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018552.070571:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332998016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018552.070581:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018552.070584:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018552.070586:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018552.070590:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2814 00000100:00080000:0.0:1734018552.070593:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018552.070634:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018552.070640:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018552.070650:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332998144/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018552.070659:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018552.070663:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018552.070665:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018552.070668:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2814 00000100:00080000:0.0:1734018552.070671:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018552.070676:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018552.070681:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018552.070683:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332997888:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018552.070690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332998272/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018552.070700:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018552.070703:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018552.070705:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018552.070708:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2814 00000100:00080000:0.0:1734018552.070711:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018552.070715:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018552.070720:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018552.070729:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332998400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018552.070737:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018552.070741:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018552.070742:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018552.070746:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2814 00000100:00080000:0.0:1734018552.070749:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018552.070753:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018552.070757:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018552.070760:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018552.070765:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332998528/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018552.070773:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018552.070778:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018552.070779:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018552.070783:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2814 00000100:00100000:1.0:1734018552.070784:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018552/real 1734018552] req@ffff8800a705d180 x1818247332997888/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018552.070787:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018552.070790:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018552.070794:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018552.070805:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332998656/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018552.070811:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332997888/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018552.070823:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018552.070827:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018552.070831:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332997888:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018552.070840:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332998016:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018552.070858:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018552.070865:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018552/real 1734018552] req@ffff8800a705c380 x1818247332998016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018552.070876:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332998016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018552.070884:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018552.070887:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018552.070890:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332998016:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018552.070897:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332998144:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018552.070911:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018552.070917:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018552/real 1734018552] req@ffff8800a705f800 x1818247332998144/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018552.070927:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332998144/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018552.070934:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018552.070937:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018552.070941:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332998144:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018552.070968:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332998272:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018552.070982:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018552.070988:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018552/real 1734018552] req@ffff8800a705d880 x1818247332998272/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018552.070998:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332998272/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018552.071006:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018552.071008:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018552.071011:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332998272:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018552.071017:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332998400:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018552.071029:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018552.071035:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018552/real 1734018552] req@ffff8800a705ca80 x1818247332998400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018552.071045:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332998400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018552.071052:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018552.071055:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018552.071058:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332998400:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018552.071064:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332998528:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018552.071080:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018552.071086:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018552/real 1734018552] req@ffff8800a705df80 x1818247332998528/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018552.071096:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332998528/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018552.071103:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018552.071106:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018552.071108:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332998528:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018552.071114:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332998656:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018552.071126:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018552.071132:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018552/real 1734018552] req@ffff8800a705c700 x1818247332998656/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018552.071142:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332998656/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018552.071150:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018552.071152:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018552.071155:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332998656:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018552.810512:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018553.812443:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018554.814473:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018555.816458:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018556.818501:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018557.078346:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018557.078355:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018557.078359:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018557.078371:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2819 00000100:00080000:0.0:1734018557.078381:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018557.078389:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018557.078425:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247332998784/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018557.078461:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018557.078467:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018557.078470:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018557.078477:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2819 00000100:00080000:0.0:1734018557.078482:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018557.078488:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018557.078501:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734018557.078564:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332998784:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018557.078569:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247332998912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018557.078586:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018557.078590:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018557.078593:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018557.078598:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2819 00000100:00080000:0.0:1734018557.078603:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018557.078608:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018557.078615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018557.078631:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247332999040/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018557.078637:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018557.078644:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018557.078649:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018557.078652:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018557.078657:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2819 00000100:00080000:0.0:1734018557.078661:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018557.078661:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018557/real 1734018557] req@ffff8800a705e680 x1818247332998784/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018557.078667:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018557.078673:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018557.078686:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247332999168/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018557.078695:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247332998784/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018557.078699:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018557.078705:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018557.078707:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734018557.078709:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018557.078713:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2819 00000100:00080000:1.0:1734018557.078713:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:0.0:1734018557.078717:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018557.078720:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332998784:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018557.078723:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018557.078729:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018557.078740:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247332999296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018557.078747:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332998912:192.168.201.152@tcp:38:kworker.0 00000100:00080000:0.0:1734018557.078755:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018557.078759:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018557.078762:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018557.078767:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2819 00000100:00080000:0.0:1734018557.078771:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018557.078773:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018557.078777:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018557.078784:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018557.078784:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018557/real 1734018557] req@ffff8800a705d500 x1818247332998912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018557.078797:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247332999424/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018557.078802:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247332998912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018557.078810:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018557.078815:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1734018557.078815:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018557.078818:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734018557.078819:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00080000:0.0:1734018557.078823:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2819 00000100:00100000:1.0:1734018557.078825:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332998912:192.168.201.152@tcp:38:kworker.0 00000100:00080000:0.0:1734018557.078827:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018557.078832:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018557.078833:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332999040:192.168.201.152@tcp:8:kworker.0 00000008:00100000:0.0:1734018557.078839:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018557.078851:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247332999552/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018557.078855:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018557.078864:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018557/real 1734018557] req@ffff8800a705c700 x1818247332999040/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018557.078882:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247332999040/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018557.078894:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018557.078898:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018557.078903:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332999040:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018557.078912:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332999168:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018557.078938:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018557.078950:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018557/real 1734018557] req@ffff8800a705df80 x1818247332999168/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018557.078963:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247332999168/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018557.078971:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018557.078974:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018557.078977:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332999168:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018557.078994:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332999296:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018557.079011:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018557.079017:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018557/real 1734018557] req@ffff8800a705ca80 x1818247332999296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018557.079028:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247332999296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018557.079036:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018557.079038:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018557.079041:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332999296:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018557.079045:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332999424:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018557.079060:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018557.079066:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018557/real 1734018557] req@ffff8800a705d880 x1818247332999424/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018557.079076:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247332999424/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018557.079084:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018557.079086:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018557.079089:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332999424:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018557.079094:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332999552:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018557.079105:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018557.079111:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018557/real 1734018557] req@ffff8800a705f800 x1818247332999552/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018557.079121:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247332999552/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018557.079128:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018557.079130:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018557.079133:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247332999552:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018557.820443:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018558.822524:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018559.824527:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018560.826490:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018561.828515:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018562.086356:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018562.086365:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018562.086370:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018562.086381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2824 00000100:00080000:0.0:1734018562.086391:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018562.086399:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018562.086469:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247332999680/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018562.086503:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018562.086507:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018562.086509:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018562.086514:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2824 00000100:00080000:0.0:1734018562.086517:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018562.086521:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018562.086531:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018562.086542:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247332999808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018562.086558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018562.086562:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018562.086564:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018562.086568:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2824 00000100:00080000:0.0:1734018562.086570:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018562.086574:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018562.086580:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018562.086588:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247332999936/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018562.086597:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018562.086600:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018562.086602:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018562.086605:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2824 00000100:00080000:0.0:1734018562.086607:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018562.086611:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018562.086615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018562.086623:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333000064/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018562.086624:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332999680:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018562.086632:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018562.086635:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018562.086636:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018562.086640:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2824 00000100:00080000:0.0:1734018562.086644:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018562.086647:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018562.086651:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018562.086659:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333000192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018562.086668:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018562.086671:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018562.086673:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018562.086677:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2824 00000100:00080000:0.0:1734018562.086680:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018562.086683:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018562.086688:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018562.086691:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018562.086695:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333000320/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018562.086703:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018562.086706:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018562.086708:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018562.086711:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2824 00000100:00100000:1.0:1734018562.086713:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018562/real 1734018562] req@ffff8800a705c380 x1818247332999680/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018562.086714:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018562.086717:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018562.086721:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018562.086728:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333000448/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018562.086740:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247332999680/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018562.086750:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018562.086753:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018562.086758:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247332999680:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018562.086766:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332999808:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018562.086783:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018562.086789:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018562/real 1734018562] req@ffff8800a705d180 x1818247332999808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018562.086800:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247332999808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018562.086808:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018562.086810:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018562.086814:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332999808:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018562.086819:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332999936:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018562.086833:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018562.086838:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018562/real 1734018562] req@ffff8800a705ed80 x1818247332999936/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018562.086848:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247332999936/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018562.086856:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018562.086859:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018562.086862:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247332999936:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018562.086887:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333000064:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018562.086900:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018562.086906:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018562/real 1734018562] req@ffff8800a705e300 x1818247333000064/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018562.086917:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333000064/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018562.086925:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018562.086927:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018562.086930:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333000064:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018562.086935:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333000192:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018562.086947:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018562.086953:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018562/real 1734018562] req@ffff8800a705c000 x1818247333000192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018562.086963:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333000192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018562.086970:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018562.086972:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018562.086975:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333000192:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018562.086981:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333000320:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018562.086997:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018562.087003:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018562/real 1734018562] req@ffff8800a705dc00 x1818247333000320/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018562.087014:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333000320/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018562.087021:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018562.087023:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018562.087026:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333000320:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018562.087031:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333000448:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018562.087042:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018562.087048:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018562/real 1734018562] req@ffff8800a705f100 x1818247333000448/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018562.087059:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333000448/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018562.087066:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018562.087068:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018562.087071:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333000448:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018562.830487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018563.832484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018564.834515:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018565.836489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018566.838521:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018567.094332:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018567.094376:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018567.094381:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018567.094392:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2829 00000100:00080000:0.0:1734018567.094402:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018567.094410:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018567.094445:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333000576/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018567.094484:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018567.094489:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018567.094491:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018567.094495:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2829 00000100:00080000:0.0:1734018567.094499:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018567.094502:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018567.094512:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018567.094523:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333000704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018567.094533:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018567.094536:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018567.094538:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018567.094541:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2829 00000100:00080000:0.0:1734018567.094544:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018567.094547:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018567.094552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018567.094562:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333000832/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018567.094570:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018567.094573:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018567.094575:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018567.094578:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2829 00000100:00080000:0.0:1734018567.094580:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018567.094584:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018567.094589:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018567.094593:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333000576:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018567.094597:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333000960/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018567.094606:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018567.094609:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018567.094610:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018567.094614:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2829 00000100:00080000:0.0:1734018567.094616:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018567.094620:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018567.094624:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018567.094631:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333001088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018567.094639:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018567.094643:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018567.094645:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018567.094648:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2829 00000100:00080000:0.0:1734018567.094651:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018567.094654:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018567.094658:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018567.094663:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018567.094666:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333001216/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018567.094674:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018567.094678:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018567.094679:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018567.094682:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2829 00000100:00080000:0.0:1734018567.094685:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018567.094685:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018567/real 1734018567] req@ffff8800a705ea00 x1818247333000576/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018567.094689:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018567.094693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018567.094700:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333001344/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018567.094713:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333000576/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018567.094724:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018567.094727:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018567.094732:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333000576:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018567.094739:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333000704:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018567.094755:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018567.094762:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018567/real 1734018567] req@ffff8800a705fb80 x1818247333000704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018567.094773:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333000704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018567.094781:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018567.094784:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018567.094787:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333000704:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018567.094793:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333000832:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018567.094806:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018567.094812:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018567/real 1734018567] req@ffff8800a705f100 x1818247333000832/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018567.094821:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333000832/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018567.094829:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018567.094832:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018567.094835:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333000832:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018567.094860:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333000960:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018567.094873:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018567.094879:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018567/real 1734018567] req@ffff8800a705dc00 x1818247333000960/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018567.094890:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333000960/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018567.094898:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018567.094900:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018567.094903:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333000960:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018567.094909:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333001088:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018567.094920:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018567.094926:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018567/real 1734018567] req@ffff8800a705c000 x1818247333001088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018567.094936:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333001088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018567.094943:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018567.094945:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018567.094948:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333001088:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018567.094954:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333001216:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018567.094969:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018567.094975:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018567/real 1734018567] req@ffff8800a705e300 x1818247333001216/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018567.094985:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333001216/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018567.094992:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018567.094995:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018567.094998:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333001216:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018567.095002:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333001344:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018567.095014:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018567.095020:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018567/real 1734018567] req@ffff8800a705ed80 x1818247333001344/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018567.095029:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333001344/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018567.095037:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018567.095039:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018567.095042:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333001344:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018567.840536:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018568.842574:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018569.844520:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018570.846520:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018571.848447:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018572.102355:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018572.102365:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018572.102369:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018572.102381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2834 00000100:00080000:0.0:1734018572.102392:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018572.102400:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018572.102437:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333001472/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018572.102471:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018572.102475:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018572.102478:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018572.102482:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2834 00000100:00080000:0.0:1734018572.102486:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018572.102490:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018572.102499:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018572.102510:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333001600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018572.102521:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018572.102524:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018572.102526:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018572.102529:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2834 00000100:00080000:0.0:1734018572.102532:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018572.102535:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018572.102540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018572.102548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333001728/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018572.102556:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018572.102559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018572.102561:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018572.102564:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2834 00000100:00080000:0.0:1734018572.102567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018572.102570:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018572.102574:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018572.102582:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333001856/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018572.102587:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333001472:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018572.102591:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018572.102594:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018572.102596:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018572.102599:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2834 00000100:00080000:0.0:1734018572.102602:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018572.102605:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018572.102610:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018572.102617:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333001984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018572.102625:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018572.102628:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018572.102630:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018572.102633:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2834 00000100:00080000:0.0:1734018572.102645:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018572.102649:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018572.102653:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018572.102654:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018572.102663:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333002112/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018572.102671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018572.102674:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018572.102675:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018572.102677:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018572/real 1734018572] req@ffff8800a705d180 x1818247333001472/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018572.102679:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2834 00000100:00080000:0.0:1734018572.102682:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018572.102685:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018572.102690:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018572.102697:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333002240/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018572.102704:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333001472/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018572.102715:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018572.102718:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018572.102723:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333001472:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018572.102730:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333001600:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018572.102746:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018572.102753:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018572/real 1734018572] req@ffff8800a705c380 x1818247333001600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018572.102763:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333001600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018572.102771:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018572.102774:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018572.102777:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333001600:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018572.102782:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333001728:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018572.102796:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018572.102802:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018572/real 1734018572] req@ffff8800a705f800 x1818247333001728/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018572.102812:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333001728/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018572.102819:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018572.102822:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018572.102825:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333001728:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018572.102853:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333001856:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018572.102866:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018572.102872:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018572/real 1734018572] req@ffff8800a705d880 x1818247333001856/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018572.102883:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333001856/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018572.102890:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018572.102892:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018572.102895:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333001856:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018572.102900:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333001984:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018572.102912:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018572.102918:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018572/real 1734018572] req@ffff8800a705ca80 x1818247333001984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018572.102927:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333001984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018572.102935:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018572.102937:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018572.102940:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333001984:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018572.102946:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333002112:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018572.102961:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018572.102967:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018572/real 1734018572] req@ffff8800a705df80 x1818247333002112/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018572.102977:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333002112/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018572.102984:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018572.102986:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018572.102989:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333002112:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018572.102994:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333002240:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018572.103006:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018572.103012:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018572/real 1734018572] req@ffff8800a705c700 x1818247333002240/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018572.103021:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333002240/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018572.103028:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018572.103030:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018572.103033:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333002240:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018572.850501:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018573.852510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018574.854478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018575.856489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018576.858515:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018577.110335:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018577.110348:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018577.110353:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018577.110368:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2839 00000100:00080000:0.0:1734018577.110379:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018577.110390:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018577.110439:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333002368/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018577.110478:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018577.110483:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018577.110485:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018577.110490:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2839 00000100:00080000:0.0:1734018577.110493:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018577.110498:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018577.110510:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018577.110522:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333002496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018577.110532:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018577.110535:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018577.110537:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018577.110541:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2839 00000100:00080000:0.0:1734018577.110544:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018577.110548:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018577.110553:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018577.110565:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333002624/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018577.110575:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018577.110578:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018577.110580:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018577.110583:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2839 00000100:00080000:0.0:1734018577.110586:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018577.110590:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018577.110595:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018577.110604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333002752/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018577.110604:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333002368:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018577.110614:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018577.110617:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018577.110619:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018577.110622:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2839 00000100:00080000:0.0:1734018577.110625:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018577.110628:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018577.110633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018577.110641:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333002880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018577.110650:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018577.110653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018577.110655:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018577.110659:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2839 00000100:00080000:0.0:1734018577.110672:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018577.110676:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018577.110681:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018577.110689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333003008/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018577.110698:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018577.110701:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018577.110703:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018577.110706:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2839 00000100:00080000:0.0:1734018577.110709:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018577.110710:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333002496:192.168.201.152@tcp:38:kworker.0 00000100:00080000:0.0:1734018577.110712:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018577.110717:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018577.110726:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333003136/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018577.110734:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333002624:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018577.110758:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333002752:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018577.110775:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333002880:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018577.110790:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333003008:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018577.110808:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333003136:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018577.860455:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018578.862479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018579.864482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734018580.116635:0:1957:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734018580.116691:0:1957:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734018580.116712:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018580.116761:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018580.116775:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018580.116785:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018580.116795:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018580.116806:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018580.116817:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734018580.116856:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018577/real 1734018580] req@ffff8800a705df80 x1818247333002752/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018580.116894:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333002752/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018580.116912:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018580.116918:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018580.116925:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333002752:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018580.116937:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018577/real 1734018580] req@ffff8800a705ca80 x1818247333002880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018580.116956:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333002880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018580.116968:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018580.116972:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018580.116978:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333002880:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018580.116988:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018577/real 1734018580] req@ffff8800a705d880 x1818247333003008/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018580.117005:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333003008/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018580.117017:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018580.117021:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018580.117027:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333003008:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018580.117036:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018577/real 1734018580] req@ffff8800a705f800 x1818247333003136/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018580.117054:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333003136/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018580.117066:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018580.117070:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018580.117075:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333003136:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018580.117084:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018577/real 1734018580] req@ffff8800a705d500 x1818247333002368/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018580.117104:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333002368/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018580.117117:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018580.117186:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018580.117192:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333002368:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018580.117203:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018577/real 1734018580] req@ffff8800a705e680 x1818247333002496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018580.117303:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333002496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018580.117327:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018580.117332:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018580.117337:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333002496:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018580.117347:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018577/real 1734018580] req@ffff8800a705c700 x1818247333002624/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018580.117364:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333002624/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018580.117376:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018580.117381:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018580.117386:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333002624:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018580.866442:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018581.868477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018582.118348:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018582.118358:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018582.118363:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018582.118373:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2844 00000100:00080000:0.0:1734018582.118383:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018582.118390:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018582.118426:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333003264/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018582.118460:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018582.118464:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018582.118466:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018582.118471:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2844 00000100:00080000:0.0:1734018582.118474:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018582.118478:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018582.118488:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018582.118499:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333003392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018582.118509:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018582.118512:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018582.118514:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018582.118518:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2844 00000100:00080000:0.0:1734018582.118520:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018582.118524:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018582.118529:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018582.118538:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333003520/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018582.118547:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018582.118550:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018582.118551:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018582.118554:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2844 00000100:00080000:0.0:1734018582.118557:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018582.118560:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018582.118565:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018582.118572:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333003648/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018582.118578:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333003264:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018582.118581:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018582.118584:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018582.118586:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018582.118590:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2844 00000100:00080000:0.0:1734018582.118592:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018582.118595:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018582.118600:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018582.118607:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333003776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018582.118616:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018582.118619:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018582.118621:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018582.118625:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2844 00000100:00080000:0.0:1734018582.118637:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018582.118640:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018582.118645:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018582.118646:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018582.118655:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333003904/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018582.118663:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018582.118666:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018582.118668:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018582.118669:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018582/real 1734018582] req@ffff8800a705c380 x1818247333003264/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018582.118671:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2844 00000100:00080000:0.0:1734018582.118675:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018582.118678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018582.118683:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018582.118688:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333003264/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018582.118690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333004032/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018582.118699:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018582.118702:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018582.118707:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333003264:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018582.118714:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333003392:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018582.118731:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018582.118738:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018582/real 1734018582] req@ffff8800a705d180 x1818247333003392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018582.118749:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333003392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018582.118757:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018582.118759:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018582.118763:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333003392:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018582.118768:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333003520:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018582.118781:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018582.118787:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018582/real 1734018582] req@ffff8800a705ed80 x1818247333003520/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018582.118797:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333003520/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018582.118805:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018582.118807:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018582.118810:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333003520:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018582.118838:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333003648:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018582.118851:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018582.118857:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018582/real 1734018582] req@ffff8800a705e300 x1818247333003648/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018582.118868:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333003648/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018582.118882:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018582.118885:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018582.118888:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333003648:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018582.118893:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333003776:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018582.118905:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018582.118911:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018582/real 1734018582] req@ffff8800a705c000 x1818247333003776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018582.118920:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333003776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018582.118928:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018582.118930:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018582.118933:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333003776:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018582.118939:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333003904:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018582.118950:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018582.118956:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018582/real 1734018582] req@ffff8800a705dc00 x1818247333003904/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018582.118966:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333003904/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018582.118973:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018582.118975:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018582.118978:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333003904:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018582.118984:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333004032:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018582.118996:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018582.119001:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018582/real 1734018582] req@ffff8800a705f100 x1818247333004032/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018582.119011:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333004032/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018582.119018:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018582.119020:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018582.119023:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333004032:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018582.870470:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018583.872516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018584.874495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018585.876463:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018586.878502:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018587.126367:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018587.126376:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018587.126380:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018587.126391:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2849 00000100:00080000:0.0:1734018587.126401:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018587.126408:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018587.126444:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333004160/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018587.126486:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018587.126491:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018587.126493:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018587.126497:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2849 00000100:00080000:0.0:1734018587.126500:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018587.126504:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018587.126514:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018587.126525:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333004288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018587.126535:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018587.126538:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018587.126540:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018587.126543:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2849 00000100:00080000:0.0:1734018587.126546:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018587.126549:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018587.126554:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018587.126564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333004416/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018587.126572:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018587.126575:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018587.126577:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018587.126580:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2849 00000100:00080000:0.0:1734018587.126583:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018587.126587:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018587.126591:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018587.126594:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333004160:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018587.126599:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333004544/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018587.126608:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018587.126611:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018587.126613:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018587.126616:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2849 00000100:00080000:0.0:1734018587.126619:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018587.126622:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018587.126627:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018587.126634:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333004672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018587.126643:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018587.126646:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018587.126647:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018587.126650:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2849 00000100:00080000:0.0:1734018587.126653:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018587.126656:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018587.126661:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018587.126661:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018587.126668:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333004800/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018587.126676:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018587.126680:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018587.126682:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018587.126683:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018587/real 1734018587] req@ffff8800a705fb80 x1818247333004160/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018587.126686:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2849 00000100:00080000:0.0:1734018587.126689:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018587.126692:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018587.126696:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018587.126703:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333004160/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018587.126704:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333004928/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018587.126713:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018587.126716:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018587.126721:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333004160:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018587.126728:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333004288:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018587.126745:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018587.126752:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018587/real 1734018587] req@ffff8800a705ea00 x1818247333004288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018587.126771:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333004288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018587.126779:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018587.126781:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018587.126785:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333004288:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018587.126790:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333004416:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018587.126803:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018587.126809:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018587/real 1734018587] req@ffff8800a705f100 x1818247333004416/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018587.126819:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333004416/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018587.126826:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018587.126829:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018587.126832:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333004416:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018587.126856:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333004544:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018587.126868:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018587.126874:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018587/real 1734018587] req@ffff8800a705dc00 x1818247333004544/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018587.126885:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333004544/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018587.126892:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018587.126895:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018587.126898:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333004544:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018587.126904:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333004672:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018587.126916:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018587.126922:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018587/real 1734018587] req@ffff8800a705c000 x1818247333004672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018587.126932:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333004672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018587.126940:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018587.126942:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018587.126945:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333004672:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018587.126950:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333004800:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018587.126962:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018587.126968:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018587/real 1734018587] req@ffff8800a705e300 x1818247333004800/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018587.126977:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333004800/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018587.126984:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018587.126986:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018587.126989:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333004800:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018587.126995:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333004928:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018587.127066:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018587.127074:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018587/real 1734018587] req@ffff8800a705ed80 x1818247333004928/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018587.127085:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333004928/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018587.127093:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018587.127095:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018587.127098:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333004928:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018587.880505:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018588.882481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018589.884477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018590.886528:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018591.888447:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018592.134339:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018592.134350:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018592.134356:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018592.134370:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2854 00000100:00080000:0.0:1734018592.134383:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018592.134392:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018592.134441:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333005056/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018592.134479:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018592.134484:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018592.134486:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018592.134492:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2854 00000100:00080000:0.0:1734018592.134495:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018592.134499:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018592.134511:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018592.134523:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333005184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018592.134535:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018592.134539:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018592.134540:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018592.134544:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2854 00000100:00080000:0.0:1734018592.134547:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018592.134550:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018592.134555:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018592.134564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333005312/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018592.134573:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018592.134576:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018592.134577:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018592.134581:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2854 00000100:00080000:0.0:1734018592.134584:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018592.134587:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018592.134592:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018592.134600:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333005440/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018592.134608:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018592.134611:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018592.134613:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018592.134617:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2854 00000100:00080000:0.0:1734018592.134620:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018592.134623:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018592.134628:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018592.134636:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333005568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018592.134641:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333005056:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018592.134646:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018592.134649:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018592.134650:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018592.134654:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2854 00000100:00080000:0.0:1734018592.134657:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018592.134660:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018592.134665:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018592.134675:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333005696/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018592.134683:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018592.134687:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018592.134688:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018592.134691:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2854 00000100:00080000:0.0:1734018592.134706:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018592.134710:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018592.134714:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018592.134721:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018592.134724:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333005824/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018592.134747:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018592/real 1734018592] req@ffff8800a705d180 x1818247333005056/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018592.134768:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333005056/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018592.134780:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018592.134784:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018592.134789:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333005056:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018592.134797:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333005184:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018592.134815:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018592.134822:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018592/real 1734018592] req@ffff8800a705c380 x1818247333005184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018592.134833:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333005184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018592.134842:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018592.134845:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018592.134848:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333005184:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018592.134855:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333005312:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018592.134868:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018592.134874:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018592/real 1734018592] req@ffff8800a705c700 x1818247333005312/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018592.134885:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333005312/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018592.134892:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018592.134895:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018592.134898:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333005312:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018592.134904:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333005440:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018592.134968:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018592.134977:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018592/real 1734018592] req@ffff8800a705e680 x1818247333005440/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018592.134987:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333005440/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018592.134995:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018592.134998:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018592.135001:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333005440:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018592.135032:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333005568:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018592.135048:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018592.135055:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018592/real 1734018592] req@ffff8800a705d500 x1818247333005568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018592.135066:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333005568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018592.135074:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018592.135076:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018592.135079:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333005568:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018592.135085:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333005696:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018592.135098:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018592.135104:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018592/real 1734018592] req@ffff8800a705f800 x1818247333005696/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018592.135115:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333005696/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018592.135122:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018592.135124:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018592.135127:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333005696:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018592.135133:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333005824:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018592.135144:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018592.135150:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018592/real 1734018592] req@ffff8800a705d880 x1818247333005824/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018592.135161:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333005824/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018592.135169:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018592.135171:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018592.135174:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333005824:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018592.890498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018593.892465:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018594.894464:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018595.896451:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018596.898477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018597.142362:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018597.142374:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018597.142379:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018597.142393:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2859 00000100:00080000:0.0:1734018597.142406:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018597.142415:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018597.142463:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333005952/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018597.142501:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018597.142505:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018597.142507:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018597.142512:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2859 00000100:00080000:0.0:1734018597.142516:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018597.142520:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018597.142532:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018597.142544:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333006080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018597.142555:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018597.142558:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018597.142560:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018597.142564:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2859 00000100:00080000:0.0:1734018597.142567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018597.142571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018597.142577:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018597.142589:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333006208/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018597.142599:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018597.142602:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018597.142604:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018597.142608:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2859 00000100:00100000:1.0:1734018597.142609:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333005952:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018597.142611:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018597.142615:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018597.142620:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018597.142629:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333006336/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018597.142637:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018597.142641:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018597.142642:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018597.142646:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2859 00000100:00080000:0.0:1734018597.142648:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018597.142652:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018597.142657:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018597.142665:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333006464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018597.142673:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018597.142676:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018597.142679:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018597.142684:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2859 00000100:00080000:0.0:1734018597.142687:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018597.142691:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018597.142695:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018597.142701:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018597.142703:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333006592/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018597.142712:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018597.142715:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018597.142716:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018597.142720:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2859 00000100:00080000:0.0:1734018597.142722:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018597.142726:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018597.142729:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018597/real 1734018597] req@ffff8800a705ca80 x1818247333005952/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018597.142730:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018597.142739:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333006720/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018597.142749:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333005952/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018597.142762:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018597.142765:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018597.142770:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333005952:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018597.142780:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333006080:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018597.142803:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018597.142810:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018597/real 1734018597] req@ffff8800a705df80 x1818247333006080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018597.142821:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333006080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018597.142875:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018597.142880:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018597.142884:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333006080:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018597.142912:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333006208:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018597.142932:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018597.142940:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018597/real 1734018597] req@ffff8800a705d880 x1818247333006208/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018597.142951:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333006208/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018597.142959:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018597.142962:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018597.142965:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333006208:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018597.142971:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333006336:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018597.142984:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018597.142989:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018597/real 1734018597] req@ffff8800a705f800 x1818247333006336/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018597.142999:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333006336/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018597.143007:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018597.143009:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018597.143013:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333006336:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018597.143019:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333006464:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018597.143036:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018597.143042:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018597/real 1734018597] req@ffff8800a705d500 x1818247333006464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018597.143052:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333006464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018597.143060:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018597.143063:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018597.143066:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333006464:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018597.143072:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333006592:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018597.143085:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018597.143090:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018597/real 1734018597] req@ffff8800a705e680 x1818247333006592/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018597.143100:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333006592/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018597.143107:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018597.143110:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018597.143113:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333006592:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018597.143118:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333006720:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018597.143132:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018597.143138:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018597/real 1734018597] req@ffff8800a705c700 x1818247333006720/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018597.143154:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333006720/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018597.143161:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018597.143164:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018597.143167:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333006720:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018597.900510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018598.902506:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018599.904475:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018600.906475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018601.908486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018602.150358:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018602.150369:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018602.150375:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018602.150389:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2864 00000100:00080000:0.0:1734018602.150400:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018602.150409:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018602.150458:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333006848/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018602.150504:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018602.150509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018602.150511:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018602.150516:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2864 00000100:00080000:0.0:1734018602.150519:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018602.150524:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018602.150536:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018602.150548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333006976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018602.150558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018602.150561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018602.150563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018602.150567:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2864 00000100:00080000:0.0:1734018602.150570:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018602.150574:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018602.150578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018602.150587:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333007104/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018602.150596:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018602.150599:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018602.150601:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018602.150604:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2864 00000100:00080000:0.0:1734018602.150607:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018602.150611:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018602.150616:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018602.150624:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333007232/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018602.150634:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018602.150637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018602.150639:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018602.150643:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2864 00000100:00100000:1.0:1734018602.150643:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333006848:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018602.150645:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018602.150649:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018602.150654:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018602.150663:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333007360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018602.150671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018602.150675:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018602.150676:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018602.150680:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2864 00000100:00080000:0.0:1734018602.150683:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018602.150687:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018602.150691:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018602.150699:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333007488/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018602.150708:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018602.150712:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018602.150714:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018602.150717:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2864 00000100:00080000:0.0:1734018602.150721:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018602.150721:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018602.150724:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018602.150729:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018602.150739:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333007616/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018602.150783:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018602/real 1734018602] req@ffff8800a705c380 x1818247333006848/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018602.150805:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333006848/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018602.150817:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018602.150821:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018602.150826:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333006848:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018602.150834:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333006976:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018602.150856:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018602.150864:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018602/real 1734018602] req@ffff8800a705d180 x1818247333006976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018602.150874:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333006976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018602.150883:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018602.150886:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018602.150890:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333006976:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018602.150897:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333007104:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018602.150911:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018602.150917:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018602/real 1734018602] req@ffff8800a705ed80 x1818247333007104/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018602.150927:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333007104/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018602.150935:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018602.150938:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018602.150941:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333007104:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018602.150969:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333007232:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018602.150981:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018602.150987:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018602/real 1734018602] req@ffff8800a705e300 x1818247333007232/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018602.150998:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333007232/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018602.151006:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018602.151008:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018602.151011:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333007232:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018602.151017:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333007360:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018602.151033:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018602.151039:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018602/real 1734018602] req@ffff8800a705c000 x1818247333007360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018602.151049:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333007360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018602.151057:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018602.151059:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018602.151062:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333007360:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018602.151068:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333007488:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018602.151080:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018602.151086:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018602/real 1734018602] req@ffff8800a705dc00 x1818247333007488/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018602.151097:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333007488/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018602.151104:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018602.151107:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018602.151110:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333007488:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018602.151116:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333007616:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018602.151128:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018602.151134:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018602/real 1734018602] req@ffff8800a705f100 x1818247333007616/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018602.151151:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333007616/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018602.151158:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018602.151161:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018602.151164:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333007616:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018602.910476:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018603.912499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018604.914498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018605.916474:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018606.918500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018607.158367:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018607.158378:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018607.158382:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018607.158393:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2869 00000100:00080000:0.0:1734018607.158403:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018607.158411:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018607.158448:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333007744/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018607.158482:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018607.158486:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018607.158488:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018607.158493:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2869 00000100:00080000:0.0:1734018607.158496:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018607.158500:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018607.158509:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018607.158521:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333007872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018607.158532:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018607.158535:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018607.158537:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018607.158540:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2869 00000100:00080000:0.0:1734018607.158543:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018607.158546:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018607.158551:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018607.158561:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333008000/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018607.158569:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018607.158572:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018607.158574:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018607.158577:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2869 00000100:00080000:0.0:1734018607.158579:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018607.158583:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018607.158587:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018607.158595:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333008128/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018607.158597:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333007744:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018607.158604:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018607.158608:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018607.158609:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018607.158612:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2869 00000100:00080000:0.0:1734018607.158618:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018607.158621:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018607.158625:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018607.158633:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333008256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018607.158641:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018607.158644:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018607.158646:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018607.158649:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2869 00000100:00080000:0.0:1734018607.158652:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018607.158655:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018607.158696:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734018607.158699:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018607.158709:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333008384/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018607.158717:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018607.158720:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018607/real 1734018607] req@ffff8800a705ea00 x1818247333007744/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018607.158721:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018607.158722:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018607.158726:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2869 00000100:00080000:0.0:1734018607.158729:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018607.158732:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018607.158737:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018607.158739:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333007744/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018607.158745:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333008512/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018607.158750:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018607.158754:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018607.158759:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333007744:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018607.158766:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333007872:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018607.158786:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018607.158792:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018607/real 1734018607] req@ffff8800a705fb80 x1818247333007872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018607.158804:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333007872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018607.158812:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018607.158815:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018607.158819:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333007872:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018607.158825:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333008000:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018607.158838:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018607.158844:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018607/real 1734018607] req@ffff8800a705f100 x1818247333008000/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018607.158854:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333008000/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018607.158862:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018607.158864:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018607.158868:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333008000:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018607.158891:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333008128:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018607.158904:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018607.158910:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018607/real 1734018607] req@ffff8800a705dc00 x1818247333008128/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018607.158920:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333008128/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018607.158928:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018607.158930:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018607.158933:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333008128:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018607.158938:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333008256:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018607.158954:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018607.158960:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018607/real 1734018607] req@ffff8800a705c000 x1818247333008256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018607.158970:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333008256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018607.158978:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018607.158980:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018607.158984:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333008256:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018607.158989:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333008384:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018607.159001:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018607.159006:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018607/real 1734018607] req@ffff8800a705e300 x1818247333008384/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018607.159016:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333008384/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018607.159023:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018607.159025:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018607.159028:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333008384:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018607.159034:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333008512:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018607.159045:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018607.159051:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018607/real 1734018607] req@ffff8800a705ed80 x1818247333008512/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018607.159067:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333008512/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018607.159075:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018607.159077:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018607.159080:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333008512:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018607.920497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018608.922527:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018609.924491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018610.926534:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018611.928473:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018612.166486:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018612.166498:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018612.166504:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018612.166518:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2874 00000100:00080000:2.0:1734018612.166531:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018612.166541:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018612.166637:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333008640/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018612.166684:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018612.166689:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018612.166692:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018612.166698:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2874 00000100:00080000:2.0:1734018612.166701:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018612.166706:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018612.166719:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018612.166731:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333008768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018612.166743:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018612.166747:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018612.166749:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018612.166753:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2874 00000100:00080000:2.0:1734018612.166755:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018612.166759:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018612.166764:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018612.166773:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333008896/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018612.166781:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018612.166784:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018612.166786:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018612.166790:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2874 00000100:00080000:2.0:1734018612.166792:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018612.166796:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018612.166801:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018612.166810:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333008640:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734018612.166813:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247333009024/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018612.166824:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018612.166828:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018612.166830:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018612.166833:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2874 00000100:00080000:2.0:1734018612.166836:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018612.166840:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018612.166844:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018612.166853:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247333009152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018612.166861:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018612.166865:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018612.166866:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018612.166870:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2874 00000100:00080000:2.0:1734018612.166874:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018612.166877:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018612.166882:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018612.166883:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018612.166890:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247333009280/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018612.166899:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018612.166903:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018612.166904:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018612.166907:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018612/real 1734018612] req@ffff8800a73c9880 x1818247333008640/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018612.166908:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2874 00000100:00080000:2.0:1734018612.166912:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018612.166915:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018612.166920:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018612.166927:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333008640/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018612.166929:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333009408/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018612.166939:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018612.166943:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018612.166948:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333008640:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018612.166956:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333008768:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018612.166974:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018612.166981:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018612/real 1734018612] req@ffff8800a73cb100 x1818247333008768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018612.166992:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333008768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018612.167000:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018612.167004:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018612.167007:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333008768:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018612.167015:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333008896:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018612.167029:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018612.167035:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018612/real 1734018612] req@ffff8800a73cad80 x1818247333008896/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018612.167045:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333008896/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018612.167053:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018612.167056:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018612.167059:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333008896:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018612.167085:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333009024:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018612.167098:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018612.167104:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018612/real 1734018612] req@ffff8800a73c9c00 x1818247333009024/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018612.167114:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247333009024/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018612.167122:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018612.167125:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018612.167128:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333009024:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018612.167133:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333009152:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018612.167149:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018612.167155:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018612/real 1734018612] req@ffff8800a73c9500 x1818247333009152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018612.167165:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247333009152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018612.167173:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018612.167176:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018612.167179:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333009152:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018612.167185:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333009280:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018612.167197:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018612.167203:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018612/real 1734018612] req@ffff8800a73c8000 x1818247333009280/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018612.167213:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247333009280/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018612.167270:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018612.167272:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018612.167275:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333009280:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018612.167282:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333009408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018612.167299:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018612.167306:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018612/real 1734018612] req@ffff8800a73c8e00 x1818247333009408/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018612.167323:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333009408/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018612.167331:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018612.167334:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018612.167336:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333009408:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018612.930528:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018613.932469:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018614.934513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018615.936507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018616.938558:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018617.174587:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018617.174598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018617.174603:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018617.174614:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2879 00000100:00080000:0.0:1734018617.174625:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018617.174633:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018617.174671:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333009536/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018617.174712:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018617.174718:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018617.174722:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018617.174729:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2879 00000100:00080000:0.0:1734018617.174734:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018617.174740:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018617.174752:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018617.174771:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333009664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018617.174786:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333009536:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018617.174787:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018617.174792:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018617.174795:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018617.174800:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2879 00000100:00080000:0.0:1734018617.174804:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018617.174809:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018617.174817:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018617.174829:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333009792/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018617.174842:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018617.174847:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018617.174849:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018617.174854:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2879 00000800:00000100:1.0:1734018617.174858:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018617.174859:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018617.174863:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018617.174870:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018617.174882:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333009920/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018617.174882:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018617/real 1734018617] req@ffff8800a705d180 x1818247333009536/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018617.174897:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018617.174902:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018617.174904:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018617.174905:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333009536/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018617.174910:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2879 00000100:00080000:0.0:1734018617.174915:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018617.174920:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:1.0:1734018617.174920:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018617.174924:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000008:00100000:0.0:1734018617.174926:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734018617.174931:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333009536:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018617.174938:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333010048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018617.174951:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018617.174957:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1734018617.174957:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333009664:192.168.201.152@tcp:38:kworker.0 00000100:00080000:0.0:1734018617.174960:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018617.174965:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2879 00000100:00080000:0.0:1734018617.174969:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018617.174974:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018617.174981:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018617.174982:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018617.174993:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333010176/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018617.174993:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018617/real 1734018617] req@ffff8800a705c380 x1818247333009664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018617.175008:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018617.175011:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333009664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018617.175014:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018617.175016:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018617.175021:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2879 00000100:00080000:1.0:1734018617.175024:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018617.175025:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1734018617.175028:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00080000:0.0:1734018617.175030:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018617.175033:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333009664:192.168.201.152@tcp:38:kworker.0 00000008:00100000:0.0:1734018617.175037:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018617.175042:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333009792:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734018617.175050:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333010304/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018617.175063:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018617.175073:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018617/real 1734018617] req@ffff8800a705c700 x1818247333009792/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018617.175090:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333009792/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018617.175102:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018617.175106:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018617.175111:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333009792:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018617.175120:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333009920:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018617.175146:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018617.175158:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018617/real 1734018617] req@ffff8800a705e680 x1818247333009920/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018617.175169:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333009920/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018617.175177:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018617.175180:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018617.175183:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333009920:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018617.175188:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333010048:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018617.175206:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018617.175212:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018617/real 1734018617] req@ffff8800a705d500 x1818247333010048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018617.175270:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333010048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018617.175278:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018617.175280:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018617.175284:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333010048:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018617.175302:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333010176:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018617.175319:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018617.175326:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018617/real 1734018617] req@ffff8800a705f800 x1818247333010176/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018617.175336:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333010176/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018617.175344:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018617.175346:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018617.175349:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333010176:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018617.175354:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333010304:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018617.175366:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018617.175371:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018617/real 1734018617] req@ffff8800a705d880 x1818247333010304/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018617.175388:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333010304/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018617.175395:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018617.175397:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018617.175400:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333010304:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018617.940397:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018618.942543:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018619.944513:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018620.946502:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018621.948502:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018622.182495:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018622.182504:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018622.182510:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018622.182524:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2884 00000100:00080000:2.0:1734018622.182536:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018622.182546:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018622.182594:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247333010432/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018622.182637:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018622.182642:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018622.182643:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018622.182649:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2884 00000100:00080000:2.0:1734018622.182652:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018622.182657:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018622.182669:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018622.182682:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247333010560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018622.182692:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018622.182696:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018622.182698:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018622.182702:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2884 00000100:00080000:2.0:1734018622.182704:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018622.182708:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018622.182713:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018622.182725:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247333010688/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018622.182733:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018622.182736:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018622.182738:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018622.182741:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2884 00000100:00080000:2.0:1734018622.182744:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018622.182747:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018622.182752:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018622.182763:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333010816/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018622.182765:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333010432:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734018622.182773:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018622.182776:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018622.182778:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018622.182782:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2884 00000100:00080000:2.0:1734018622.182785:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018622.182789:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018622.182794:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018622.182803:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247333010944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018622.182812:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018622.182815:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018622.182816:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018622.182821:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2884 00000100:00080000:2.0:1734018622.182824:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018622.182828:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018622.182832:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018622.182840:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247333011072/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018622.182841:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:2.0:1734018622.182849:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018622.182852:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018622.182854:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018622.182857:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2884 00000100:00080000:2.0:1734018622.182861:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018622.182864:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018622.182865:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018622/real 1734018622] req@ffff8800a73ca680 x1818247333010432/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734018622.182869:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018622.182877:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247333011200/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018622.182885:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ca680 x1818247333010432/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018622.182897:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018622.182900:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018622.182905:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333010432:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018622.182913:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333010560:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018622.182932:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018622.182938:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018622/real 1734018622] req@ffff8800a73c9f80 x1818247333010560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018622.182950:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9f80 x1818247333010560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018622.182957:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018622.182961:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018622.182964:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333010560:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018622.182971:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333010688:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018622.182985:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018622.182991:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018622/real 1734018622] req@ffff8800a73caa00 x1818247333010688/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018622.183001:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73caa00 x1818247333010688/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018622.183009:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018622.183012:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018622.183015:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333010688:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018622.183041:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333010816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018622.183054:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018622.183060:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018622/real 1734018622] req@ffff8800a73c8e00 x1818247333010816/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018622.183071:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333010816/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018622.183078:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018622.183081:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018622.183084:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333010816:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018622.183090:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333010944:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018622.183106:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018622.183112:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018622/real 1734018622] req@ffff8800a73c8000 x1818247333010944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018622.183122:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247333010944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018622.183130:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018622.183132:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018622.183135:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333010944:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018622.183141:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333011072:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018622.183153:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018622.183159:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018622/real 1734018622] req@ffff8800a73c9500 x1818247333011072/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018622.183169:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247333011072/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018622.183177:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018622.183179:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018622.183182:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333011072:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018622.183188:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333011200:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018622.183200:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018622.183206:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018622/real 1734018622] req@ffff8800a73c9c00 x1818247333011200/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018622.183271:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247333011200/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018622.183279:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018622.183282:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018622.183285:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333011200:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018622.950469:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018623.952524:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018624.954518:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018625.956501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018626.958531:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018627.190528:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018627.190539:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018627.190544:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018627.190555:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2889 00000100:00080000:3.0:1734018627.190565:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018627.190573:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018627.190608:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333011328/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018627.190649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018627.190653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018627.190655:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018627.190659:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2889 00000100:00080000:3.0:1734018627.190663:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018627.190667:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018627.190677:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018627.190690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333011456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018627.190700:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018627.190703:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018627.190705:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018627.190709:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2889 00000100:00080000:3.0:1734018627.190711:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018627.190714:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018627.190719:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018627.190727:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333011584/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018627.190736:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018627.190739:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018627.190740:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018627.190744:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2889 00000100:00080000:3.0:1734018627.190746:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018627.190750:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018627.190754:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018627.190762:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333011712/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018627.190764:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333011328:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734018627.190771:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018627.190775:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018627.190776:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018627.190779:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2889 00000100:00080000:3.0:1734018627.190782:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018627.190785:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018627.190789:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018627.190797:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333011840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018627.190805:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018627.190808:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018627.190810:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018627.190813:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2889 00000100:00080000:3.0:1734018627.190826:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018627.190829:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018627.190832:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734018627.190834:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018627.190842:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333011968/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018627.190850:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018627.190854:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734018627.190855:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018627/real 1734018627] req@ffff8800a73cea00 x1818247333011328/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018627.190856:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018627.190860:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2889 00000100:00080000:3.0:1734018627.190863:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018627.190866:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018627.190870:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018627.190874:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333011328/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018627.190878:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333012096/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018627.190885:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018627.190889:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018627.190893:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333011328:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018627.190901:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333011456:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018627.190918:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018627.190924:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018627/real 1734018627] req@ffff8800a73ce680 x1818247333011456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018627.190935:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333011456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018627.190943:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018627.190946:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018627.190950:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333011456:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018627.190956:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333011584:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018627.190969:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018627.190975:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018627/real 1734018627] req@ffff8800a73cce00 x1818247333011584/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018627.190985:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333011584/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018627.190993:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018627.190995:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018627.190999:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333011584:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018627.191024:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333011712:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018627.191037:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018627.191042:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018627/real 1734018627] req@ffff8800a73ced80 x1818247333011712/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018627.191053:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333011712/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018627.191061:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018627.191063:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018627.191066:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333011712:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018627.191072:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333011840:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018627.191087:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018627.191092:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018627/real 1734018627] req@ffff8800a73cd880 x1818247333011840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018627.191103:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333011840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018627.191110:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018627.191112:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018627.191115:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333011840:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018627.191120:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333011968:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018627.191132:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018627.191138:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018627/real 1734018627] req@ffff8800a73cf480 x1818247333011968/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018627.191147:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333011968/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018627.191154:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018627.191156:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018627.191159:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333011968:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018627.191165:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333012096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018627.191177:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018627.191183:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018627/real 1734018627] req@ffff8800a73cca80 x1818247333012096/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018627.191199:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333012096/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018627.191207:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018627.191209:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018627.191212:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333012096:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018627.960516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018628.962525:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018629.964485:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018630.966528:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018631.968491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018632.198376:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018632.198388:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018632.198394:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018632.198408:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2894 00000100:00080000:3.0:1734018632.198419:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018632.198429:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018632.198477:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333012224/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018632.198516:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018632.198520:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018632.198522:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018632.198528:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2894 00000100:00080000:3.0:1734018632.198531:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018632.198535:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018632.198548:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018632.198561:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333012352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018632.198571:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018632.198575:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018632.198577:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018632.198581:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2894 00000100:00080000:3.0:1734018632.198583:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018632.198587:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018632.198592:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018632.198601:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333012480/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018632.198611:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018632.198614:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018632.198615:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018632.198619:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2894 00000100:00080000:3.0:1734018632.198622:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018632.198623:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333012224:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734018632.198625:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018632.198630:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018632.198638:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333012608/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018632.198647:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018632.198650:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018632.198653:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018632.198657:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2894 00000100:00080000:3.0:1734018632.198659:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018632.198663:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018632.198667:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018632.198676:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333012736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018632.198684:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018632.198687:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018632.198688:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018632.198693:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2894 00000100:00080000:3.0:1734018632.198696:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018632.198699:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018632.198703:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018632.198708:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734018632.198711:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333012864/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018632.198719:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018632.198723:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018632.198724:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018632.198727:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2894 00000100:00080000:3.0:1734018632.198730:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018632.198733:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018632.198736:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018632/real 1734018632] req@ffff8800a73cd500 x1818247333012224/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734018632.198738:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018632.198748:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333012992/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018632.198758:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333012224/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018632.198770:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018632.198773:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018632.198778:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333012224:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018632.198786:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333012352:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018632.198804:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018632.198811:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018632/real 1734018632] req@ffff8800a73cc380 x1818247333012352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018632.198822:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333012352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018632.198830:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018632.198834:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018632.198838:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333012352:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018632.198866:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333012480:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018632.198882:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018632.198888:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018632/real 1734018632] req@ffff8800a73cf100 x1818247333012480/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018632.198900:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333012480/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018632.198907:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018632.198910:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018632.198913:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333012480:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018632.198918:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333012608:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018632.198929:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018632.198935:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018632/real 1734018632] req@ffff8800a73cc700 x1818247333012608/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018632.198945:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333012608/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018632.198952:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018632.198954:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018632.198957:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333012608:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018632.198963:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333012736:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018632.198980:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018632.198987:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018632/real 1734018632] req@ffff8800a73cdf80 x1818247333012736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018632.198997:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333012736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018632.199005:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018632.199007:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018632.199010:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333012736:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018632.199015:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333012864:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018632.199028:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018632.199034:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018632/real 1734018632] req@ffff8800a73cf800 x1818247333012864/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018632.199044:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333012864/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018632.199052:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018632.199054:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018632.199057:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333012864:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018632.199063:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333012992:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018632.199075:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018632.199081:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018632/real 1734018632] req@ffff8800a73cfb80 x1818247333012992/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018632.199098:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333012992/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018632.199107:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018632.199109:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018632.199112:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333012992:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018632.970486:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018633.972505:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018634.974497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018635.976491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018636.978522:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018637.206327:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018637.206338:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018637.206343:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018637.206358:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2899 00000100:00080000:3.0:1734018637.206372:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018637.206382:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018637.206432:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333013120/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018637.206470:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018637.206474:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018637.206476:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018637.206481:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2899 00000100:00080000:3.0:1734018637.206485:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018637.206489:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018637.206501:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018637.206515:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333013248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018637.206525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018637.206529:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018637.206531:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018637.206535:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2899 00000100:00080000:3.0:1734018637.206540:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018637.206544:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018637.206549:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018637.206558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333013376/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018637.206566:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018637.206570:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018637.206571:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018637.206575:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2899 00000100:00080000:3.0:1734018637.206577:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018637.206581:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018637.206585:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018637.206589:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333013120:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734018637.206593:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333013504/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018637.206602:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018637.206606:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018637.206607:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018637.206611:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2899 00000100:00080000:3.0:1734018637.206614:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018637.206617:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018637.206622:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018637.206630:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333013632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018637.206645:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018637.206649:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018637.206651:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018637.206655:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2899 00000100:00080000:3.0:1734018637.206661:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018637.206664:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018637.206669:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018637.206674:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734018637.206677:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333013760/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018637.206685:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018637.206688:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018637.206689:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018637.206693:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2899 00000100:00080000:3.0:1734018637.206696:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018637.206700:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018637.206701:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018637/real 1734018637] req@ffff8800a73cdc00 x1818247333013120/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734018637.206704:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018637.206713:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333013888/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018637.206723:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333013120/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018637.206735:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018637.206738:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018637.206743:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333013120:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018637.206751:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333013248:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018637.206770:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018637.206777:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018637/real 1734018637] req@ffff8800a73cfb80 x1818247333013248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018637.206788:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333013248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018637.206797:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018637.206800:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018637.206804:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333013248:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018637.206810:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333013376:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018637.206825:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018637.206831:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018637/real 1734018637] req@ffff8800a73cf800 x1818247333013376/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018637.206841:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333013376/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018637.206849:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018637.206852:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018637.206856:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333013376:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018637.206886:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333013504:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018637.206899:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018637.206905:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018637/real 1734018637] req@ffff8800a73cdf80 x1818247333013504/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018637.206916:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333013504/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018637.206925:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018637.206927:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018637.206930:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333013504:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018637.206935:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333013632:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018637.206952:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018637.206958:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018637/real 1734018637] req@ffff8800a73cc700 x1818247333013632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018637.206968:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333013632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018637.206975:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018637.206978:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018637.206980:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333013632:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018637.206986:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333013760:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018637.206998:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018637.207004:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018637/real 1734018637] req@ffff8800a73cf100 x1818247333013760/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018637.207015:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333013760/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018637.207022:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018637.207025:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018637.207028:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333013760:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018637.207033:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333013888:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018637.207046:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018637.207052:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018637/real 1734018637] req@ffff8800a73cc380 x1818247333013888/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018637.207068:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333013888/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018637.207076:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018637.207078:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018637.207081:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333013888:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018637.980517:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018638.982464:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018639.984468:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018640.986479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018641.988424:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018642.214350:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018642.214361:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018642.214368:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018642.214382:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2904 00000100:00080000:3.0:1734018642.214394:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018642.214404:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018642.214452:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333014016/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018642.214489:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018642.214494:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018642.214496:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018642.214501:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2904 00000100:00080000:3.0:1734018642.214505:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018642.214510:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018642.214521:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018642.214532:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333014144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018642.214543:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018642.214547:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018642.214548:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018642.214552:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2904 00000100:00080000:3.0:1734018642.214555:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018642.214558:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018642.214563:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018642.214572:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333014272/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018642.214589:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018642.214592:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018642.214593:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018642.214597:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2904 00000100:00080000:3.0:1734018642.214600:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018642.214601:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333014016:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734018642.214603:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018642.214607:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018642.214615:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333014400/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018642.214623:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018642.214627:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018642.214628:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018642.214632:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2904 00000100:00080000:3.0:1734018642.214634:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018642.214638:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018642.214642:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018642.214650:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333014528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018642.214659:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018642.214662:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018642.214664:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018642.214669:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2904 00000100:00080000:3.0:1734018642.214674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018642.214677:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018642.214682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018642.214690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333014656/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018642.214697:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018642.214701:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018642.214702:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018642.214706:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2904 00000100:00080000:3.0:1734018642.214708:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018642.214712:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018642.214716:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018642.214717:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333014144:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734018642.214727:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333014784/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018642.214749:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333014272:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018642.214767:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333014400:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018642.214782:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333014528:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018642.214799:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333014656:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018642.214815:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333014784:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018642.990501:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018643.992490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018644.994473:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734018645.220653:0:1958:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734018645.220720:0:1958:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734018645.220748:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018645.220816:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018645.220830:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018645.220843:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018645.220854:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018645.220865:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018645.220879:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734018645.220906:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018642/real 1734018645] req@ffff8800a73cf480 x1818247333014272/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018645.220958:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333014272/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018645.220978:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018645.220985:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018645.220993:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333014272:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018645.221005:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018642/real 1734018645] req@ffff8800a73cd880 x1818247333014400/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018645.221077:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333014400/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018645.221091:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018645.221096:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018645.221102:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333014400:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018645.221112:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018642/real 1734018645] req@ffff8800a73ced80 x1818247333014528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018645.221131:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333014528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018645.221144:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018645.221148:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018645.221153:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333014528:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018645.221163:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018642/real 1734018645] req@ffff8800a73cce00 x1818247333014656/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018645.221182:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333014656/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018645.221195:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018645.221199:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018645.221206:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333014656:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018645.221337:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018642/real 1734018645] req@ffff8800a73ce680 x1818247333014784/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018645.221360:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333014784/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018645.221373:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018645.221377:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018645.221382:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333014784:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018645.221392:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018642/real 1734018645] req@ffff8800a73cd500 x1818247333014016/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018645.221411:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333014016/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018645.221423:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018645.221427:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018645.221432:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333014016:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018645.221443:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018642/real 1734018645] req@ffff8800a73cca80 x1818247333014144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018645.221460:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333014144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018645.221473:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018645.221478:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018645.221483:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333014144:192.168.201.152@tcp:38:kworker.0 00010000:00010000:3.0:1734018645.996446:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018646.998511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018647.222363:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018647.222374:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018647.222379:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018647.222394:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2909 00000100:00080000:3.0:1734018647.222406:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018647.222416:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018647.222464:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333014912/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018647.222509:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018647.222514:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018647.222516:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018647.222521:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2909 00000100:00080000:3.0:1734018647.222525:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018647.222529:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018647.222541:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018647.222555:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333015040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018647.222565:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018647.222568:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018647.222570:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018647.222574:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2909 00000100:00080000:3.0:1734018647.222577:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018647.222581:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018647.222586:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018647.222595:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333015168/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018647.222604:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018647.222607:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018647.222609:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018647.222613:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2909 00000100:00080000:3.0:1734018647.222615:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018647.222617:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333014912:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734018647.222620:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018647.222624:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018647.222632:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333015296/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018647.222640:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018647.222644:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018647.222645:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018647.222649:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2909 00000100:00080000:3.0:1734018647.222651:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018647.222655:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018647.222659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018647.222666:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333015424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018647.222675:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018647.222678:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018647.222679:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018647.222683:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2909 00000100:00080000:3.0:1734018647.222695:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018647.222699:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018647.222704:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018647.222705:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734018647.222712:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333015552/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018647.222721:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018647.222725:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018647.222727:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018647.222731:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2909 00000100:00100000:1.0:1734018647.222732:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018647/real 1734018647] req@ffff8800a73cea00 x1818247333014912/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018647.222734:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018647.222738:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018647.222742:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018647.222750:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333015680/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018647.222754:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333014912/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018647.222766:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018647.222769:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018647.222774:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333014912:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018647.222781:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333015040:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018647.222800:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018647.222807:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018647/real 1734018647] req@ffff8800a73cca80 x1818247333015040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018647.222818:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333015040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018647.222825:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018647.222828:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018647.222831:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333015040:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018647.222858:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333015168:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018647.222873:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018647.222879:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018647/real 1734018647] req@ffff8800a73cd500 x1818247333015168/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018647.222889:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333015168/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018647.222898:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018647.222901:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018647.222904:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333015168:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018647.222909:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333015296:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018647.222921:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018647.222927:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018647/real 1734018647] req@ffff8800a73ce680 x1818247333015296/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018647.222937:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333015296/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018647.222944:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018647.222946:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018647.222949:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333015296:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018647.222955:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333015424:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018647.222968:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018647.222974:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018647/real 1734018647] req@ffff8800a73cce00 x1818247333015424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018647.223035:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333015424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018647.223043:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018647.223046:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018647.223050:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333015424:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018647.223056:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333015552:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018647.223073:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018647.223080:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018647/real 1734018647] req@ffff8800a73ced80 x1818247333015552/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018647.223091:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333015552/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018647.223099:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018647.223101:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018647.223105:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333015552:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018647.223110:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333015680:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018647.223122:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018647.223128:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018647/real 1734018647] req@ffff8800a73cd880 x1818247333015680/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018647.223138:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333015680/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018647.223145:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018647.223148:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018647.223152:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333015680:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018648.000453:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018649.002529:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018650.004454:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018651.006469:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018652.008498:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018652.230340:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018652.230351:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018652.230356:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018652.230370:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2914 00000100:00080000:3.0:1734018652.230382:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018652.230391:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018652.230439:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333015808/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018652.230478:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018652.230483:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018652.230485:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018652.230490:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2914 00000100:00080000:3.0:1734018652.230493:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018652.230498:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018652.230511:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018652.230523:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333015936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018652.230541:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018652.230545:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018652.230547:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018652.230551:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2914 00000100:00080000:3.0:1734018652.230553:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018652.230557:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018652.230562:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018652.230571:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333016064/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018652.230579:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018652.230582:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018652.230584:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018652.230587:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2914 00000100:00080000:3.0:1734018652.230590:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018652.230595:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018652.230600:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018652.230601:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333015808:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734018652.230608:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333016192/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018652.230617:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018652.230620:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018652.230623:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018652.230627:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2914 00000100:00080000:3.0:1734018652.230630:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018652.230633:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018652.230638:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018652.230646:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333016320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018652.230655:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018652.230658:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018652.230659:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018652.230664:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2914 00000100:00080000:3.0:1734018652.230666:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018652.230670:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018652.230674:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018652.230682:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333016448/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018652.230688:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734018652.230689:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018652.230693:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018652.230694:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018652.230698:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2914 00000100:00080000:3.0:1734018652.230701:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018652.230704:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018652.230709:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018652.230715:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018652/real 1734018652] req@ffff8800a73cf480 x1818247333015808/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018652.230719:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333016576/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018652.230738:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333015808/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018652.230750:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018652.230754:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018652.230759:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333015808:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018652.230766:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333015936:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018652.230785:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018652.230792:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018652/real 1734018652] req@ffff8800a73cc380 x1818247333015936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018652.230803:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333015936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018652.230811:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018652.230814:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018652.230817:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333015936:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018652.230823:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333016064:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018652.230837:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018652.230843:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018652/real 1734018652] req@ffff8800a73cf100 x1818247333016064/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018652.230853:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333016064/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018652.230861:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018652.230865:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018652.230868:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333016064:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018652.230945:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333016192:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018652.230964:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018652.230972:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018652/real 1734018652] req@ffff8800a73cc700 x1818247333016192/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018652.230983:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333016192/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018652.230991:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018652.230993:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018652.230996:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333016192:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018652.231002:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333016320:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018652.231014:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018652.231020:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018652/real 1734018652] req@ffff8800a73cdf80 x1818247333016320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018652.231030:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333016320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018652.231038:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018652.231040:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018652.231044:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333016320:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018652.231049:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333016448:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018652.231065:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018652.231071:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018652/real 1734018652] req@ffff8800a73cf800 x1818247333016448/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018652.231081:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333016448/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018652.231088:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018652.231090:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018652.231093:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333016448:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018652.231098:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333016576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018652.231110:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018652.231116:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018652/real 1734018652] req@ffff8800a73cfb80 x1818247333016576/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018652.231125:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333016576/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018652.231132:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018652.231134:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018652.231137:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333016576:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734018653.010513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018654.012478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018655.014492:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018656.016490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734018657.018455:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018657.238359:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018657.238370:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018657.238374:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018657.238385:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2919 00000100:00080000:3.0:1734018657.238394:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018657.238403:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018657.238441:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333016704/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018657.238478:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018657.238482:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018657.238484:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018657.238489:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2919 00000100:00080000:3.0:1734018657.238492:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018657.238496:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018657.238505:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018657.238519:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333016832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018657.238529:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018657.238532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018657.238534:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018657.238538:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2919 00000100:00080000:3.0:1734018657.238542:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018657.238546:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018657.238551:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018657.238559:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333016960/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018657.238568:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018657.238571:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018657.238572:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018657.238576:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2919 00000100:00080000:3.0:1734018657.238578:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018657.238581:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018657.238585:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018657.238594:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333017088/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018657.238599:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333016704:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734018657.238603:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018657.238607:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018657.238608:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018657.238612:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2919 00000100:00080000:3.0:1734018657.238614:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018657.238618:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018657.238622:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018657.238630:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333017216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018657.238645:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018657.238648:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018657.238650:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018657.238654:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2919 00000100:00080000:3.0:1734018657.238657:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018657.238660:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018657.238664:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018657.238671:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333017344/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018657.238679:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018657.238682:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018657.238684:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018657.238687:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2919 00000100:00080000:3.0:1734018657.238690:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018657.238692:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734018657.238693:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018657.238698:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018657.238705:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333017472/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018657.238721:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018657/real 1734018657] req@ffff8800a73cdc00 x1818247333016704/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018657.238753:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333016704/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018657.238766:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018657.238770:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018657.238775:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333016704:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018657.238783:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333016832:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018657.238802:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018657.238850:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018657/real 1734018657] req@ffff8800a73cfb80 x1818247333016832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018657.238863:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333016832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018657.238871:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018657.238875:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018657.238879:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333016832:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018657.238886:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333016960:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018657.238904:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018657.238911:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018657/real 1734018657] req@ffff8800a73cf800 x1818247333016960/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018657.238921:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333016960/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018657.238930:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018657.238934:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018657.238937:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333016960:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018657.238968:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333017088:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018657.238982:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018657.238988:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018657/real 1734018657] req@ffff8800a73cdf80 x1818247333017088/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018657.238999:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333017088/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018657.239006:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018657.239009:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018657.239012:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333017088:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018657.239017:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333017216:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018657.239030:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018657.239036:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018657/real 1734018657] req@ffff8800a73cc700 x1818247333017216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018657.239046:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333017216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018657.239053:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018657.239056:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018657.239059:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333017216:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018657.239065:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333017344:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018657.239082:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018657.239088:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018657/real 1734018657] req@ffff8800a73cf100 x1818247333017344/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018657.239099:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333017344/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018657.239106:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018657.239108:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018657.239111:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333017344:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018657.239117:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333017472:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018657.239130:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018657.239135:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018657/real 1734018657] req@ffff8800a73cc380 x1818247333017472/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018657.239145:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333017472/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018657.239152:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018657.239155:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018657.239158:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333017472:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018658.020475:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018659.022720:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018660.024615:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018661.026629:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018662.028517:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018662.246357:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018662.246367:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018662.246372:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018662.246383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2924 00000100:00080000:3.0:1734018662.246393:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018662.246402:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018662.246438:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333017600/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018662.246474:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018662.246479:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018662.246480:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018662.246485:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2924 00000100:00080000:3.0:1734018662.246489:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018662.246493:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018662.246502:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018662.246513:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333017728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018662.246524:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018662.246527:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018662.246529:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018662.246532:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2924 00000100:00080000:3.0:1734018662.246535:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018662.246538:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018662.246543:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018662.246551:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333017856/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018662.246560:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018662.246563:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018662.246565:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018662.246569:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2924 00000100:00080000:3.0:1734018662.246571:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018662.246575:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018662.246579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018662.246586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333017984/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018662.246596:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018662.246600:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018662.246601:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018662.246604:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2924 00000100:00080000:3.0:1734018662.246607:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018662.246611:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018662.246615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734018662.246620:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333017600:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734018662.246623:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333018112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018662.246632:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018662.246635:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018662.246637:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018662.246641:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2924 00000100:00080000:3.0:1734018662.246644:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018662.246647:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018662.246652:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018662.246659:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333018240/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018662.246667:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018662.246670:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018662.246672:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018662.246675:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2924 00000100:00080000:3.0:1734018662.246678:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018662.246681:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018662.246685:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018662.246694:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333018368/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018662.246714:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018662.246773:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018662/real 1734018662] req@ffff8800a73cf480 x1818247333017600/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018662.246804:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333017600/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018662.246817:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018662.246820:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018662.246826:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333017600:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018662.246835:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333017728:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018662.246859:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018662.246866:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018662/real 1734018662] req@ffff8800a73cd880 x1818247333017728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018662.246878:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333017728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018662.246886:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018662.246888:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018662.246892:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333017728:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018662.246898:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333017856:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018662.246914:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018662.246920:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018662/real 1734018662] req@ffff8800a73ced80 x1818247333017856/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018662.246930:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333017856/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018662.246938:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018662.246941:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018662.246945:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333017856:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018662.246977:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333017984:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018662.246990:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018662.246997:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018662/real 1734018662] req@ffff8800a73cce00 x1818247333017984/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018662.247008:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333017984/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018662.247015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018662.247018:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018662.247021:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333017984:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018662.247027:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333018112:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018662.247039:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018662.247045:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018662/real 1734018662] req@ffff8800a73ce680 x1818247333018112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018662.247055:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333018112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018662.247062:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018662.247065:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018662.247068:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333018112:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018662.247074:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333018240:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018662.247091:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018662.247097:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018662/real 1734018662] req@ffff8800a73cd500 x1818247333018240/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018662.247108:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333018240/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018662.247115:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018662.247117:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018662.247120:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333018240:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018662.247126:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333018368:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018662.247139:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018662.247144:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018662/real 1734018662] req@ffff8800a73cca80 x1818247333018368/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018662.247154:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333018368/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018662.247162:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018662.247165:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018662.247168:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333018368:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018663.030433:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018664.032470:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018665.034513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018666.036459:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018667.038489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018667.254332:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018667.254342:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018667.254347:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018667.254361:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2929 00000100:00080000:3.0:1734018667.254373:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018667.254383:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018667.254433:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333018496/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018667.254471:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018667.254478:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018667.254481:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018667.254489:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2929 00000100:00080000:3.0:1734018667.254494:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018667.254502:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018667.254517:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734018667.254539:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333018496:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734018667.254539:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333018624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018667.254555:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018667.254560:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018667.254562:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018667.254568:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2929 00000100:00080000:3.0:1734018667.254572:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018667.254577:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018667.254583:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018667.254596:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333018752/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018667.254608:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018667.254613:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018667.254615:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018667.254620:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2929 00000100:00080000:3.0:1734018667.254624:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018667.254629:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018667.254631:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734018667.254637:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018667.254682:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333018880/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018667.254690:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018667/real 1734018667] req@ffff8800a73cea00 x1818247333018496/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018667.254696:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018667.254702:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018667.254705:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018667.254711:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2929 00000100:00080000:3.0:1734018667.254716:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018667.254721:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018667.254726:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333018496/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734018667.254728:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018667.254740:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333019008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018667.254742:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018667.254747:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018667.254754:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333018496:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734018667.254755:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018667.254760:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018667.254762:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018667.254767:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2929 00000100:00080000:3.0:1734018667.254771:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018667.254776:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018667.254783:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018667.254787:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333018624:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734018667.254795:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333019136/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018667.254809:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018667.254815:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1734018667.254817:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734018667.254818:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018667.254824:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2929 00000100:00100000:1.0:1734018667.254828:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018667/real 1734018667] req@ffff8800a73cca80 x1818247333018624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018667.254828:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018667.254834:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018667.254841:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018667.254846:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333018624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018667.254854:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333019264/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018667.254859:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018667.254863:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018667.254868:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333018624:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018667.254877:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333018752:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018667.254903:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018667.254915:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018667/real 1734018667] req@ffff8800a73cd500 x1818247333018752/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018667.254932:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333018752/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018667.254946:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018667.254950:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018667.254956:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333018752:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018667.254965:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333018880:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018667.254988:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018667.254996:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018667/real 1734018667] req@ffff8800a73ce680 x1818247333018880/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018667.255007:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333018880/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018667.255014:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018667.255017:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018667.255020:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333018880:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018667.255025:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333019008:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018667.255037:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018667.255043:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018667/real 1734018667] req@ffff8800a73cce00 x1818247333019008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018667.255053:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333019008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018667.255060:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018667.255062:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018667.255066:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333019008:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018667.255084:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333019136:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018667.255101:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018667.255107:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018667/real 1734018667] req@ffff8800a73ced80 x1818247333019136/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018667.255118:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333019136/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018667.255125:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018667.255127:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018667.255130:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333019136:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018667.255135:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333019264:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018667.255146:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018667.255152:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018667/real 1734018667] req@ffff8800a73cd880 x1818247333019264/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018667.255161:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333019264/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018667.255168:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018667.255170:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018667.255173:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333019264:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018668.040500:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018669.042503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018670.044463:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018671.046489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018672.048487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018672.262355:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018672.262364:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018672.262370:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018672.262383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2934 00000100:00080000:3.0:1734018672.262397:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018672.262407:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018672.262455:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333019392/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018672.262492:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018672.262497:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018672.262500:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018672.262505:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2934 00000100:00080000:3.0:1734018672.262509:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018672.262513:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018672.262525:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018672.262537:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333019520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018672.262547:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018672.262551:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018672.262579:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018672.262584:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2934 00000100:00080000:3.0:1734018672.262588:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018672.262592:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018672.262597:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018672.262607:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333019648/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018672.262617:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018672.262621:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018672.262623:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018672.262625:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333019392:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734018672.262627:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2934 00000100:00080000:3.0:1734018672.262630:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018672.262633:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018672.262638:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018672.262646:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333019776/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018672.262656:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018672.262660:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018672.262661:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018672.262665:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2934 00000100:00080000:3.0:1734018672.262668:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018672.262671:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018672.262676:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018672.262684:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333019904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018672.262692:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018672.262695:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018672.262697:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018672.262701:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2934 00000100:00080000:3.0:1734018672.262704:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018672.262708:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018672.262709:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734018672.262712:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018672.262720:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333020032/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018672.262728:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018672.262731:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018672.262733:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018672.262736:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018672/real 1734018672] req@ffff8800a73cf480 x1818247333019392/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018672.262736:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2934 00000100:00080000:3.0:1734018672.262740:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018672.262743:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018672.262748:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018672.262756:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333020160/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018672.262766:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333019392/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018672.262777:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018672.262781:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018672.262786:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333019392:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018672.262794:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333019520:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018672.262812:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018672.262819:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018672/real 1734018672] req@ffff8800a73cc380 x1818247333019520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018672.262830:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333019520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018672.262838:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018672.262840:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018672.262844:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333019520:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018672.262869:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333019648:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018672.262884:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018672.262890:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018672/real 1734018672] req@ffff8800a73cf100 x1818247333019648/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018672.262900:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333019648/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018672.262909:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018672.262913:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018672.262917:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333019648:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018672.262923:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333019776:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018672.262935:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018672.262941:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018672/real 1734018672] req@ffff8800a73cc700 x1818247333019776/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018672.262950:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333019776/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018672.262958:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018672.262960:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018672.262964:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333019776:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018672.262970:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333019904:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018672.262982:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018672.262988:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018672/real 1734018672] req@ffff8800a73cdf80 x1818247333019904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018672.262998:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333019904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018672.263006:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018672.263009:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018672.263012:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333019904:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018672.263017:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333020032:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018672.263033:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018672.263039:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018672/real 1734018672] req@ffff8800a73cf800 x1818247333020032/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018672.263050:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333020032/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018672.263057:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018672.263059:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018672.263062:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333020032:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018672.263067:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333020160:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018672.263079:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018672.263085:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018672/real 1734018672] req@ffff8800a73cfb80 x1818247333020160/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018672.263095:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333020160/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018672.263102:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018672.263104:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018672.263107:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333020160:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018673.050500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018674.052507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018675.054432:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018676.056563:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018677.058548:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018677.270367:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018677.270378:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018677.270384:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018677.270399:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2939 00000100:00080000:3.0:1734018677.270412:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018677.270422:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018677.270504:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333020288/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018677.270541:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018677.270546:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018677.270548:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018677.270552:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2939 00000100:00080000:3.0:1734018677.270556:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018677.270561:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018677.270573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018677.270587:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333020416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018677.270598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018677.270602:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018677.270604:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018677.270608:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2939 00000100:00080000:3.0:1734018677.270611:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018677.270614:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018677.270620:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018677.270629:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333020544/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018677.270638:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018677.270641:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018677.270642:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018677.270646:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2939 00000100:00080000:3.0:1734018677.270648:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018677.270652:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018677.270657:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018677.270666:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333020288:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734018677.270667:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333020672/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018677.270677:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018677.270680:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018677.270682:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018677.270686:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2939 00000100:00080000:3.0:1734018677.270689:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018677.270693:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018677.270698:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018677.270707:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333020800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018677.270715:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018677.270719:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018677.270720:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018677.270725:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2939 00000100:00080000:3.0:1734018677.270731:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018677.270734:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018677.270738:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018677.270742:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734018677.270746:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333020928/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018677.270755:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018677.270758:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018677.270760:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018677.270764:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2939 00000100:00100000:1.0:1734018677.270765:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018677/real 1734018677] req@ffff8800a73cdc00 x1818247333020288/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018677.270767:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018677.270771:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018677.270775:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018677.270784:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333021056/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018677.270793:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333020288/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018677.270804:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018677.270808:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018677.270813:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333020288:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018677.270821:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333020416:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018677.270839:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018677.270846:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018677/real 1734018677] req@ffff8800a73cfb80 x1818247333020416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018677.270858:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333020416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018677.270865:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018677.270868:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018677.270872:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333020416:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018677.270878:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333020544:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018677.270892:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018677.270898:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018677/real 1734018677] req@ffff8800a73cf800 x1818247333020544/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018677.270908:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333020544/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018677.270916:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018677.270920:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018677.270923:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333020544:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018677.270949:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333020672:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018677.270963:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018677.270970:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018677/real 1734018677] req@ffff8800a73cdf80 x1818247333020672/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018677.270980:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333020672/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018677.270988:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018677.270990:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018677.270993:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333020672:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018677.270999:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333020800:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018677.271011:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018677.271017:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018677/real 1734018677] req@ffff8800a73cc700 x1818247333020800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018677.271027:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333020800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018677.271035:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018677.271037:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018677.271041:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333020800:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018677.271047:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333020928:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018677.271063:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018677.271069:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018677/real 1734018677] req@ffff8800a73cf100 x1818247333020928/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018677.271079:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333020928/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018677.271086:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018677.271088:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018677.271091:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333020928:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018677.271097:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333021056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018677.271109:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018677.271115:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018677/real 1734018677] req@ffff8800a73cc380 x1818247333021056/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018677.271125:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333021056/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018677.271132:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018677.271134:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018677.271137:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333021056:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018678.060497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018679.062509:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018680.064500:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018681.066479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018682.068480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018682.278329:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018682.278337:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018682.278341:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018682.278352:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2944 00000100:00080000:3.0:1734018682.278363:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018682.278370:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018682.278437:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333021184/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018682.278471:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018682.278477:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018682.278480:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018682.278487:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2944 00000100:00080000:3.0:1734018682.278492:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018682.278498:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018682.278511:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018682.278529:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333021312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018682.278536:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333021184:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734018682.278545:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018682.278549:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018682.278552:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018682.278558:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2944 00000100:00080000:3.0:1734018682.278561:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018682.278566:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018682.278573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018682.278584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333021440/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018682.278596:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018682.278601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018682.278603:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734018682.278608:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734018682.278608:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2944 00000100:00080000:3.0:1734018682.278612:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018682.278618:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018682.278624:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018682.278632:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018682/real 1734018682] req@ffff8800a73cf480 x1818247333021184/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018682.278636:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333021568/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018682.278649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018682.278655:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018682.278657:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018682.278662:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2944 00000100:00080000:3.0:1734018682.278666:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018682.278667:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333021184/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018682.278671:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018682.278678:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1734018682.278682:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018682.278686:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734018682.278690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333021696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018682.278692:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333021184:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734018682.278704:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018682.278709:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018682.278711:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018682.278716:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2944 00000100:00100000:1.0:1734018682.278719:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333021312:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734018682.278720:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018682.278726:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018682.278732:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018682.278744:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734018682.278745:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333021824/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018682.278756:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018682/real 1734018682] req@ffff8800a73cd880 x1818247333021312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018682.278759:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018682.278764:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018682.278767:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018682.278772:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2944 00000100:00100000:1.0:1734018682.278773:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333021312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018682.278777:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018682.278782:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:1.0:1734018682.278786:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1734018682.278788:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734018682.278790:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018682.278795:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333021312:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734018682.278802:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333021952/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018682.278803:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333021440:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018682.278825:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018682.278836:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018682/real 1734018682] req@ffff8800a73ced80 x1818247333021440/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018682.278852:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333021440/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018682.278865:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018682.278870:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018682.278875:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333021440:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018682.278885:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333021568:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018682.278910:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018682.278919:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018682/real 1734018682] req@ffff8800a73cce00 x1818247333021568/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018682.278930:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333021568/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018682.278937:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018682.278939:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018682.278943:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333021568:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018682.278948:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333021696:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018682.278963:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018682.278969:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018682/real 1734018682] req@ffff8800a73ce680 x1818247333021696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018682.278979:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333021696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018682.278986:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018682.278988:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018682.278991:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333021696:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018682.279009:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333021824:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018682.279024:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018682.279030:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018682/real 1734018682] req@ffff8800a73cd500 x1818247333021824/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018682.279041:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333021824/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018682.279048:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018682.279050:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018682.279053:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333021824:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018682.279058:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333021952:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018682.279069:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018682.279075:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018682/real 1734018682] req@ffff8800a73cca80 x1818247333021952/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018682.279085:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333021952/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018682.279092:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018682.279094:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018682.279097:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333021952:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018683.070448:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018684.072444:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018685.074512:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018686.076517:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018687.078488:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018687.286352:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018687.286363:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018687.286369:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018687.286383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2949 00000100:00080000:3.0:1734018687.286395:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018687.286405:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018687.286451:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333022080/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018687.286489:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018687.286493:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018687.286495:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018687.286500:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2949 00000100:00080000:3.0:1734018687.286504:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018687.286508:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018687.286519:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018687.286534:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333022208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018687.286544:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018687.286548:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018687.286549:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018687.286554:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2949 00000100:00080000:3.0:1734018687.286556:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018687.286560:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018687.286565:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018687.286575:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333022336/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018687.286586:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018687.286589:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018687.286591:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018687.286595:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2949 00000100:00080000:3.0:1734018687.286597:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018687.286601:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018687.286606:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333022080:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734018687.286606:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018687.286614:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333022464/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018687.286622:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018687.286626:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018687.286627:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018687.286631:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2949 00000100:00080000:3.0:1734018687.286634:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018687.286637:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018687.286641:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018687.286650:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333022592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018687.286658:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018687.286661:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018687.286662:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018687.286667:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2949 00000100:00080000:3.0:1734018687.286671:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018687.286674:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018687.286679:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734018687.286679:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018687.286687:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333022720/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018687.286695:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018687.286699:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018687.286700:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018687.286703:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018687/real 1734018687] req@ffff8800a73cea00 x1818247333022080/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018687.286704:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2949 00000100:00080000:3.0:1734018687.286707:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018687.286710:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018687.286715:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018687.286722:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333022848/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018687.286731:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333022080/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018687.286742:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018687.286746:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018687.286751:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333022080:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018687.286759:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333022208:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018687.286778:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018687.286785:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018687/real 1734018687] req@ffff8800a73cca80 x1818247333022208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018687.286796:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333022208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018687.286803:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018687.286806:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018687.286810:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333022208:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018687.286834:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333022336:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018687.286848:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018687.286854:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018687/real 1734018687] req@ffff8800a73cd500 x1818247333022336/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018687.286865:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333022336/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018687.286872:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018687.286876:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018687.286879:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333022336:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018687.286885:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333022464:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018687.286898:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018687.286904:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018687/real 1734018687] req@ffff8800a73ce680 x1818247333022464/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018687.286913:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333022464/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018687.286921:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018687.286923:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018687.286926:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333022464:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018687.286932:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333022592:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018687.286945:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018687.286951:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018687/real 1734018687] req@ffff8800a73cce00 x1818247333022592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018687.286962:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333022592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018687.286970:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018687.286972:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018687.286975:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333022592:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018687.286981:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333022720:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018687.286997:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018687.287004:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018687/real 1734018687] req@ffff8800a73ced80 x1818247333022720/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018687.287014:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333022720/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018687.287021:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018687.287023:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018687.287026:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333022720:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018687.287032:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333022848:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018687.287044:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018687.287050:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018687/real 1734018687] req@ffff8800a73cd880 x1818247333022848/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018687.287059:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333022848/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018687.287067:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018687.287069:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018687.287072:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333022848:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018688.080460:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018689.082525:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018690.084480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018691.086480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018692.088493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018692.294325:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018692.294334:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018692.294338:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018692.294348:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2954 00000100:00080000:3.0:1734018692.294358:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018692.294366:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018692.294403:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333022976/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018692.294436:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018692.294440:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018692.294442:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018692.294446:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2954 00000100:00080000:3.0:1734018692.294450:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018692.294454:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018692.294464:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018692.294475:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333023104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018692.294485:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018692.294488:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018692.294490:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018692.294493:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2954 00000100:00080000:3.0:1734018692.294496:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018692.294500:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018692.294509:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018692.294518:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333023232/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018692.294527:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018692.294530:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018692.294532:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018692.294536:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2954 00000100:00100000:1.0:1734018692.294539:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333022976:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734018692.294539:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018692.294542:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018692.294546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018692.294553:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333023360/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018692.294561:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018692.294565:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018692.294566:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018692.294569:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2954 00000100:00080000:3.0:1734018692.294572:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018692.294575:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018692.294579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018692.294586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333023488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018692.294594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018692.294597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018692.294599:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018692.294603:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2954 00000100:00080000:3.0:1734018692.294605:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018692.294606:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734018692.294609:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018692.294613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018692.294620:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333023616/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018692.294627:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018692/real 1734018692] req@ffff8800a73cf480 x1818247333022976/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018692.294629:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018692.294632:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018692.294633:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018692.294637:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2954 00000100:00080000:3.0:1734018692.294639:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018692.294643:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018692.294647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018692.294654:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333023744/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018692.294655:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333022976/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018692.294665:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018692.294668:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018692.294673:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333022976:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018692.294680:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333023104:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018692.294697:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018692.294704:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018692/real 1734018692] req@ffff8800a73cc380 x1818247333023104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018692.294715:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333023104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018692.294722:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018692.294725:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018692.294728:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333023104:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018692.294750:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333023232:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018692.294764:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018692.294770:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018692/real 1734018692] req@ffff8800a73cf100 x1818247333023232/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018692.294780:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333023232/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018692.294788:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018692.294791:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018692.294794:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333023232:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018692.294800:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333023360:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018692.294813:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018692.294819:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018692/real 1734018692] req@ffff8800a73cc700 x1818247333023360/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018692.294828:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333023360/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018692.294836:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018692.294838:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018692.294842:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333023360:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018692.294848:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333023488:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018692.294860:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018692.294865:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018692/real 1734018692] req@ffff8800a73cdf80 x1818247333023488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018692.294875:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333023488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018692.294883:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018692.294885:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018692.294888:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333023488:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018692.294894:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333023616:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018692.294909:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018692.294915:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018692/real 1734018692] req@ffff8800a73cf800 x1818247333023616/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018692.294925:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333023616/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018692.294932:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018692.294934:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018692.294937:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333023616:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018692.294943:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333023744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018692.294954:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018692.294960:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018692/real 1734018692] req@ffff8800a73cfb80 x1818247333023744/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018692.294969:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333023744/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018692.294977:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018692.294979:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018692.294982:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333023744:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018693.090465:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018694.092494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018695.094478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018696.096459:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018697.098475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018697.302329:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018697.302339:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018697.302343:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018697.302354:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2959 00000100:00080000:3.0:1734018697.302364:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018697.302372:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018697.302410:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333023872/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018697.302444:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018697.302448:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018697.302451:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018697.302456:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2959 00000100:00080000:3.0:1734018697.302459:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018697.302463:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018697.302472:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018697.302484:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333024000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018697.302494:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018697.302498:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018697.302499:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018697.302503:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2959 00000100:00080000:3.0:1734018697.302506:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018697.302509:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018697.302514:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018697.302522:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333024128/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018697.302530:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018697.302533:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018697.302535:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018697.302538:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2959 00000100:00080000:3.0:1734018697.302541:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018697.302544:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018697.302546:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333023872:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734018697.302549:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018697.302556:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333024256/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018697.302566:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018697.302569:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018697.302571:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018697.302574:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2959 00000100:00080000:3.0:1734018697.302577:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018697.302580:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018697.302584:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018697.302592:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333024384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018697.302600:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018697.302603:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018697.302605:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018697.302608:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2959 00000100:00080000:3.0:1734018697.302611:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018697.302614:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018697.302616:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734018697.302618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018697.302626:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333024512/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018697.302634:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018697.302637:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018697/real 1734018697] req@ffff8800a73cdc00 x1818247333023872/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018697.302637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018697.302638:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018697.302642:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2959 00000100:00080000:3.0:1734018697.302645:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018697.302648:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018697.302653:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018697.302660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333024640/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018697.302664:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333023872/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018697.302674:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018697.302677:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018697.302682:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333023872:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018697.302689:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333024000:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018697.302707:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018697.302713:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018697/real 1734018697] req@ffff8800a73cfb80 x1818247333024000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018697.302724:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333024000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018697.302732:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018697.302734:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018697.302738:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333024000:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018697.302743:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333024128:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018697.302757:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018697.302763:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018697/real 1734018697] req@ffff8800a73cf800 x1818247333024128/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018697.302773:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333024128/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018697.302780:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018697.302784:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018697.302787:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333024128:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018697.302811:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333024256:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018697.302823:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018697.302829:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018697/real 1734018697] req@ffff8800a73cdf80 x1818247333024256/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018697.302839:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333024256/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018697.302846:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018697.302848:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018697.302851:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333024256:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018697.302857:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333024384:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018697.302868:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018697.302874:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018697/real 1734018697] req@ffff8800a73cc700 x1818247333024384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018697.302884:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333024384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018697.302892:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018697.302894:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018697.302897:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333024384:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018697.302903:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333024512:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018697.302918:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018697.302924:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018697/real 1734018697] req@ffff8800a73cf100 x1818247333024512/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018697.302934:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333024512/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018697.302941:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018697.302943:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018697.302946:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333024512:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018697.302952:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333024640:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018697.302963:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018697.302969:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018697/real 1734018697] req@ffff8800a73cc380 x1818247333024640/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018697.302979:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333024640/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018697.302987:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018697.302989:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018697.302992:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333024640:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018698.100390:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018699.102499:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018700.104467:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018701.106441:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018702.108473:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018702.310355:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018702.310365:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018702.310371:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018702.310385:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2964 00000100:00080000:3.0:1734018702.310397:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018702.310407:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018702.310455:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333024768/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018702.310494:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018702.310499:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018702.310501:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018702.310506:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2964 00000100:00080000:3.0:1734018702.310510:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018702.310514:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018702.310526:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018702.310539:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333024896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018702.310549:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018702.310553:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018702.310555:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018702.310559:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2964 00000100:00080000:3.0:1734018702.310562:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018702.310566:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018702.310571:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018702.310580:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333025024/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018702.310589:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018702.310592:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018702.310594:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018702.310597:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2964 00000100:00080000:3.0:1734018702.310599:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018702.310603:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018702.310608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018702.310615:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333024768:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734018702.310616:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333025152/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018702.310626:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018702.310629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018702.310631:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018702.310636:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2964 00000100:00080000:3.0:1734018702.310639:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018702.310642:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018702.310647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018702.310656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333025280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018702.310664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018702.310667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018702.310669:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018702.310673:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2964 00000100:00080000:3.0:1734018702.310686:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018702.310689:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734018702.310691:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018702.310695:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018702.310703:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333025408/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018702.310712:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018702.310713:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018702/real 1734018702] req@ffff8800a73cf480 x1818247333024768/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018702.310716:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018702.310717:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018702.310721:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2964 00000100:00080000:3.0:1734018702.310725:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018702.310729:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018702.310733:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018702.310741:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333024768/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018702.310744:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333025536/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018702.310752:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018702.310756:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018702.310761:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333024768:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018702.310770:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333024896:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018702.310789:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018702.310796:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018702/real 1734018702] req@ffff8800a73cd880 x1818247333024896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018702.310807:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333024896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018702.310814:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018702.310817:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018702.310821:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333024896:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018702.310827:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333025024:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018702.310841:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018702.310847:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018702/real 1734018702] req@ffff8800a73ced80 x1818247333025024/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018702.310857:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333025024/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018702.310864:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018702.310868:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018702.310871:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333025024:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018702.310897:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333025152:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018702.310911:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018702.310917:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018702/real 1734018702] req@ffff8800a73cce00 x1818247333025152/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018702.310927:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333025152/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018702.310935:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018702.310937:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018702.310940:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333025152:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018702.310946:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333025280:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018702.310958:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018702.310964:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018702/real 1734018702] req@ffff8800a73ce680 x1818247333025280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018702.310974:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333025280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018702.310982:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018702.310984:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018702.310987:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333025280:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018702.310993:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333025408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018702.311009:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018702.311014:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018702/real 1734018702] req@ffff8800a73cd500 x1818247333025408/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018702.311025:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333025408/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018702.311032:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018702.311034:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018702.311037:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333025408:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018702.311096:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333025536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018702.311113:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018702.311120:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018702/real 1734018702] req@ffff8800a73cca80 x1818247333025536/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018702.311130:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333025536/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018702.311138:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018702.311140:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018702.311143:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333025536:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018703.110498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018704.112471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018705.114441:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018706.116487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018707.118524:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018707.318350:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018707.318360:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018707.318365:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018707.318379:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2969 00000100:00080000:3.0:1734018707.318392:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018707.318402:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018707.318452:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333025664/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018707.318491:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018707.318496:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018707.318498:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018707.318503:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2969 00000100:00080000:3.0:1734018707.318507:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018707.318511:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018707.318523:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018707.318537:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333025792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018707.318547:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018707.318551:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018707.318553:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018707.318558:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2969 00000100:00080000:3.0:1734018707.318561:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018707.318565:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018707.318570:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018707.318579:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333025920/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018707.318588:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018707.318591:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018707.318592:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018707.318596:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2969 00000100:00080000:3.0:1734018707.318598:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018707.318602:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018707.318606:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018707.318612:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333025664:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734018707.318615:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333026048/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018707.318625:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018707.318629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018707.318630:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018707.318634:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2969 00000100:00080000:3.0:1734018707.318637:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018707.318641:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018707.318645:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018707.318654:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333026176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018707.318664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018707.318667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018707.318669:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018707.318673:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2969 00000100:00080000:3.0:1734018707.318676:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018707.318680:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018707.318684:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018707.318691:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333026304/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018707.318699:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018707.318703:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018707.318704:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018707.318708:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2969 00000100:00080000:3.0:1734018707.318710:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018707.318713:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333025792:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734018707.318714:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018707.318719:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018707.318728:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333026432/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018707.318736:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333025920:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018707.318761:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333026048:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018707.318779:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333026176:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018707.318795:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333026304:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018707.318810:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333026432:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018708.120476:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018709.122495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018710.124515:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734018710.324660:0:1956:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734018710.324744:0:1956:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734018710.324776:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018710.324841:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018710.324855:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018710.324867:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018710.324880:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018710.324892:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018710.324903:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734018710.324987:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018707/real 1734018710] req@ffff8800a73ce680 x1818247333026048/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018710.325029:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333026048/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018710.325047:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018710.325053:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018710.325060:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333026048:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018710.325072:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018707/real 1734018710] req@ffff8800a73cce00 x1818247333026176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018710.325091:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333026176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018710.325104:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018710.325108:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018710.325114:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333026176:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018710.325125:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018707/real 1734018710] req@ffff8800a73ced80 x1818247333026304/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018710.325144:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333026304/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018710.325158:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018710.325162:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018710.325168:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333026304:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018710.325179:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018707/real 1734018710] req@ffff8800a73cd880 x1818247333026432/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018710.325197:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333026432/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018710.325209:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018710.325213:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018710.325318:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333026432:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018710.325331:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018707/real 1734018710] req@ffff8800a73cea00 x1818247333025664/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018710.325352:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333025664/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018710.325365:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018710.325368:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018710.325373:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333025664:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018710.325385:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018707/real 1734018710] req@ffff8800a73cca80 x1818247333025792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018710.325403:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333025792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018710.325436:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018710.325440:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018710.325446:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333025792:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018710.325457:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018707/real 1734018710] req@ffff8800a73cd500 x1818247333025920/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018710.325475:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333025920/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018710.325487:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018710.325491:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018710.325496:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333025920:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018711.126504:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018712.128481:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018712.326349:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018712.326359:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018712.326363:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018712.326375:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2974 00000100:00080000:3.0:1734018712.326385:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018712.326394:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018712.326430:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333026560/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018712.326464:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018712.326469:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018712.326471:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018712.326475:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2974 00000100:00080000:3.0:1734018712.326479:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018712.326482:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018712.326492:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018712.326503:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333026688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018712.326513:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018712.326516:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018712.326518:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018712.326522:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2974 00000100:00080000:3.0:1734018712.326524:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018712.326528:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018712.326533:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018712.326541:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333026816/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018712.326551:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018712.326554:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018712.326555:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018712.326559:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2974 00000100:00080000:3.0:1734018712.326561:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018712.326564:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018712.326569:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018712.326571:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333026560:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734018712.326577:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333026944/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018712.326585:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018712.326589:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018712.326590:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018712.326594:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2974 00000100:00080000:3.0:1734018712.326596:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018712.326600:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018712.326604:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018712.326612:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333027072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018712.326620:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018712.326623:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018712.326624:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018712.326628:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2974 00000100:00080000:3.0:1734018712.326631:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018712.326635:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018712.326638:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734018712.326639:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018712.326646:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333027200/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018712.326654:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018712.326657:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018712.326659:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018712.326660:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018712/real 1734018712] req@ffff8800a73cf480 x1818247333026560/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018712.326662:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2974 00000100:00080000:3.0:1734018712.326665:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018712.326668:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018712.326672:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018712.326679:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333026560/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018712.326680:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333027328/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018712.326690:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018712.326693:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018712.326698:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333026560:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018712.326706:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333026688:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018712.326723:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018712.326730:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018712/real 1734018712] req@ffff8800a73cc380 x1818247333026688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018712.326742:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333026688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018712.326749:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018712.326752:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018712.326755:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333026688:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018712.326761:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333026816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018712.326775:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018712.326781:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018712/real 1734018712] req@ffff8800a73cf100 x1818247333026816/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018712.326791:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333026816/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018712.326798:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018712.326800:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018712.326803:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333026816:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018712.326827:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333026944:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018712.326839:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018712.326845:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018712/real 1734018712] req@ffff8800a73cc700 x1818247333026944/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018712.326856:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333026944/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018712.326870:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018712.326916:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018712.326920:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333026944:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018712.326926:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333027072:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018712.326942:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018712.326950:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018712/real 1734018712] req@ffff8800a73cdf80 x1818247333027072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018712.326961:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333027072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018712.326969:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018712.326971:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018712.326974:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333027072:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018712.326979:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333027200:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018712.326990:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018712.326996:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018712/real 1734018712] req@ffff8800a73cf800 x1818247333027200/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018712.327006:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333027200/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018712.327013:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018712.327016:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018712.327019:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333027200:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018712.327025:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333027328:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018712.327036:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018712.327042:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018712/real 1734018712] req@ffff8800a73cfb80 x1818247333027328/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018712.327052:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333027328/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018712.327060:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018712.327062:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018712.327065:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333027328:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018713.130464:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018714.132480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018715.134468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018716.136451:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018717.138503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018717.334358:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018717.334367:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018717.334372:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018717.334382:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2979 00000100:00080000:3.0:1734018717.334393:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018717.334401:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018717.334438:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333027456/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018717.334472:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018717.334476:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018717.334478:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018717.334483:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2979 00000100:00080000:3.0:1734018717.334486:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018717.334490:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018717.334500:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018717.334512:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333027584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018717.334522:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018717.334525:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018717.334527:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018717.334531:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2979 00000100:00080000:3.0:1734018717.334533:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018717.334537:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018717.334542:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018717.334550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333027712/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018717.334558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018717.334561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018717.334563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018717.334567:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2979 00000100:00080000:3.0:1734018717.334570:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018717.334574:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018717.334578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018717.334579:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333027456:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734018717.334586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333027840/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018717.334595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018717.334598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018717.334599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018717.334603:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2979 00000100:00080000:3.0:1734018717.334605:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018717.334608:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018717.334613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018717.334620:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333027968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018717.334628:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018717.334631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018717.334633:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018717.334637:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2979 00000100:00080000:3.0:1734018717.334640:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018717.334643:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018717.334647:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734018717.334647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018717.334655:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333028096/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018717.334664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018717.334667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734018717.334669:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018717/real 1734018717] req@ffff8800a73cdc00 x1818247333027456/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018717.334669:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018717.334672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2979 00000100:00080000:3.0:1734018717.334675:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018717.334678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018717.334682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018717.334689:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333027456/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018717.334690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333028224/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018717.334700:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018717.334703:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018717.334708:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333027456:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018717.334715:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333027584:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018717.334732:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018717.334739:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018717/real 1734018717] req@ffff8800a73cfb80 x1818247333027584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018717.334758:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333027584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018717.334766:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018717.334769:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018717.334772:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333027584:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018717.334778:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333027712:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018717.334839:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018717.334847:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018717/real 1734018717] req@ffff8800a73cf800 x1818247333027712/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018717.334858:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333027712/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018717.334866:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018717.334868:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018717.334871:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333027712:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018717.334895:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333027840:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018717.334908:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018717.334915:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018717/real 1734018717] req@ffff8800a73cdf80 x1818247333027840/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018717.334925:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333027840/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018717.334932:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018717.334935:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018717.334938:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333027840:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018717.334944:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333027968:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018717.334956:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018717.334962:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018717/real 1734018717] req@ffff8800a73cc700 x1818247333027968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018717.334972:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333027968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018717.334980:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018717.334982:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018717.334985:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333027968:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018717.334990:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333028096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018717.335002:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018717.335008:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018717/real 1734018717] req@ffff8800a73cf100 x1818247333028096/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018717.335018:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333028096/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018717.335025:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018717.335027:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018717.335030:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333028096:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018717.335035:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333028224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018717.335050:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018717.335057:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018717/real 1734018717] req@ffff8800a73cc380 x1818247333028224/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018717.335067:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333028224/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018717.335074:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018717.335076:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018717.335079:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333028224:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018718.140504:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018719.142511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018720.144491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018721.146495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018722.148456:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018722.342362:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018722.342372:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018722.342378:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018722.342391:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2984 00000100:00080000:3.0:1734018722.342402:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018722.342412:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018722.342461:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333028352/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018722.342499:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018722.342503:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018722.342507:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018722.342512:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2984 00000100:00080000:3.0:1734018722.342516:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018722.342520:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018722.342533:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018722.342544:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333028480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018722.342555:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018722.342559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018722.342561:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018722.342565:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2984 00000100:00080000:3.0:1734018722.342568:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018722.342571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018722.342577:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018722.342585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333028608/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018722.342594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018722.342597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018722.342599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018722.342602:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2984 00000100:00080000:3.0:1734018722.342605:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018722.342608:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018722.342613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018722.342622:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333028352:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734018722.342622:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333028736/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018722.342633:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018722.342637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018722.342638:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018722.342642:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2984 00000100:00080000:3.0:1734018722.342645:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018722.342649:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018722.342653:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018722.342662:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333028864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018722.342670:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018722.342673:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018722.342675:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018722.342679:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2984 00000100:00080000:3.0:1734018722.342685:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018722.342688:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018722.342693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018722.342695:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734018722.342736:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333028992/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018722.342746:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018722.342749:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018722.342751:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018722.342755:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2984 00000100:00100000:1.0:1734018722.342768:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018722/real 1734018722] req@ffff8800a73cf480 x1818247333028352/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018722.342768:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018722.342772:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018722.342778:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018722.342789:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333028352/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018722.342789:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333029120/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018722.342801:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018722.342804:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018722.342810:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333028352:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018722.342819:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333028480:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018722.342841:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018722.342848:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018722/real 1734018722] req@ffff8800a73cd500 x1818247333028480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018722.342859:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333028480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018722.342867:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018722.342871:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018722.342874:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333028480:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018722.342880:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333028608:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018722.342894:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018722.342900:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018722/real 1734018722] req@ffff8800a73cca80 x1818247333028608/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018722.342910:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333028608/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018722.342918:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018722.342920:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018722.342924:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333028608:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018722.342950:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333028736:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018722.342963:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018722.342969:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018722/real 1734018722] req@ffff8800a73cea00 x1818247333028736/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018722.342980:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333028736/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018722.342987:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018722.342990:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018722.342992:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333028736:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018722.342999:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333028864:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018722.343012:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018722.343018:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018722/real 1734018722] req@ffff8800a73cd880 x1818247333028864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018722.343029:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333028864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018722.343037:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018722.343039:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018722.343042:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333028864:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018722.343048:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333028992:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018722.343060:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018722.343066:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018722/real 1734018722] req@ffff8800a73ced80 x1818247333028992/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018722.343075:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333028992/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018722.343082:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018722.343085:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018722.343088:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333028992:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018722.343094:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333029120:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018722.343106:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018722.343112:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018722/real 1734018722] req@ffff8800a73cce00 x1818247333029120/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018722.343122:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333029120/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018722.343130:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018722.343132:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018722.343135:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333029120:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018723.150468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018724.152493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018725.154494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018726.156474:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018727.158502:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018727.350375:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018727.350386:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018727.350392:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018727.350406:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2989 00000100:00080000:3.0:1734018727.350418:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018727.350427:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018727.350475:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333029248/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018727.350513:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018727.350518:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018727.350520:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018727.350525:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2989 00000100:00080000:3.0:1734018727.350529:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018727.350534:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018727.350546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018727.350561:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333029376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018727.350572:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018727.350576:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018727.350578:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018727.350582:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2989 00000100:00080000:3.0:1734018727.350585:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018727.350589:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018727.350594:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018727.350604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333029504/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018727.350615:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018727.350657:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018727.350659:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018727.350663:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2989 00000100:00080000:3.0:1734018727.350666:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018727.350667:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333029248:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734018727.350669:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018727.350674:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018727.350684:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333029632/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018727.350694:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018727.350697:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018727.350699:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018727.350704:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2989 00000100:00080000:3.0:1734018727.350707:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018727.350711:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018727.350716:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018727.350724:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333029760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018727.350732:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018727.350735:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018727.350737:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734018727.350741:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734018727.350741:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2989 00000100:00080000:3.0:1734018727.350745:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018727.350749:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018727.350754:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018727.350762:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333029888/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018727.350764:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018727/real 1734018727] req@ffff8800a73ce680 x1818247333029248/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018727.350771:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018727.350774:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018727.350776:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018727.350780:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2989 00000100:00080000:3.0:1734018727.350782:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018727.350785:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333029248/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018727.350786:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018727.350790:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734018727.350797:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1734018727.350799:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333030016/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018727.350800:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018727.350805:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333029248:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018727.350814:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333029376:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018727.350833:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018727.350840:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018727/real 1734018727] req@ffff8800a73cce00 x1818247333029376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018727.350851:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333029376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018727.350859:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018727.350863:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018727.350867:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333029376:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018727.350892:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333029504:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018727.350907:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018727.350914:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018727/real 1734018727] req@ffff8800a73ced80 x1818247333029504/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018727.350925:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333029504/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018727.350933:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018727.350936:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018727.350939:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333029504:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018727.350945:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333029632:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018727.350957:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018727.350963:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018727/real 1734018727] req@ffff8800a73cd880 x1818247333029632/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018727.350973:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333029632/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018727.350981:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018727.350983:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018727.350986:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333029632:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018727.350993:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333029760:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018727.351009:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018727.351016:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018727/real 1734018727] req@ffff8800a73cea00 x1818247333029760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018727.351026:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333029760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018727.351033:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018727.351036:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018727.351039:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333029760:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018727.351045:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333029888:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018727.351058:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018727.351064:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018727/real 1734018727] req@ffff8800a73cca80 x1818247333029888/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018727.351075:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333029888/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018727.351082:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018727.351085:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018727.351088:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333029888:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018727.351094:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333030016:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018727.351106:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018727.351113:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018727/real 1734018727] req@ffff8800a73cd500 x1818247333030016/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018727.351130:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333030016/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018727.351137:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018727.351140:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018727.351143:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333030016:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018728.160470:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018729.162496:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018730.164498:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018731.166489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018732.168494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018732.358358:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018732.358367:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018732.358371:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018732.358381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2994 00000100:00080000:3.0:1734018732.358391:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018732.358399:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018732.358433:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333030144/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018732.358467:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018732.358471:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018732.358473:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018732.358477:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2994 00000100:00080000:3.0:1734018732.358481:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018732.358485:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018732.358495:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018732.358505:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333030272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018732.358515:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018732.358519:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018732.358521:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018732.358525:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2994 00000100:00080000:3.0:1734018732.358528:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018732.358569:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018732.358575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018732.358585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333030400/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018732.358595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018732.358597:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333030144:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734018732.358599:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018732.358601:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018732.358604:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2994 00000100:00080000:3.0:1734018732.358607:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018732.358620:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018732.358625:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018732.358633:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333030528/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018732.358641:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018732.358644:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018732.358646:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018732.358649:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2994 00000100:00080000:3.0:1734018732.358653:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018732.358656:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018732.358660:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1734018732.358663:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734018732.358674:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333030656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018732.358684:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018732.358685:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018732/real 1734018732] req@ffff8800a73cf480 x1818247333030144/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018732.358687:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018732.358689:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018732.358693:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2994 00000100:00080000:3.0:1734018732.358696:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018732.358699:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018732.358703:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018732.358704:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333030144/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018732.358711:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333030784/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018732.358714:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018732.358718:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734018732.358719:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018732.358722:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333030144:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734018732.358722:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018732.358723:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018732.358727:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2994 00000100:00080000:3.0:1734018732.358729:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018732.358730:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333030272:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734018732.358733:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018732.358737:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018732.358746:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333030912/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018732.358747:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018732.358754:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018732/real 1734018732] req@ffff8800a73cc380 x1818247333030272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018732.358765:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333030272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018732.358773:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018732.358776:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018732.358780:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333030272:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018732.358803:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333030400:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018732.358817:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018732.358824:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018732/real 1734018732] req@ffff8800a73cf100 x1818247333030400/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018732.358835:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333030400/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018732.358842:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018732.358844:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018732.358847:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333030400:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018732.358853:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333030528:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018732.358865:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018732.358871:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018732/real 1734018732] req@ffff8800a73cc700 x1818247333030528/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018732.358880:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333030528/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018732.358888:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018732.358890:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018732.358893:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333030528:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018732.358899:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333030656:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018732.358915:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018732.358921:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018732/real 1734018732] req@ffff8800a73cdf80 x1818247333030656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018732.358931:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333030656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018732.358939:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018732.358941:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018732.358944:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333030656:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018732.358950:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333030784:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018732.358961:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018732.358967:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018732/real 1734018732] req@ffff8800a73cf800 x1818247333030784/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018732.358977:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333030784/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018732.358984:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018732.358986:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018732.358989:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333030784:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018732.358995:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333030912:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018732.359007:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018732.359013:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018732/real 1734018732] req@ffff8800a73cfb80 x1818247333030912/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018732.359029:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333030912/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018732.359037:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018732.359039:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018732.359042:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333030912:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018733.170468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018734.172544:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018735.174474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018736.176505:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018737.178522:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018737.366347:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018737.366357:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018737.366362:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018737.366372:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 2999 00000100:00080000:3.0:1734018737.366381:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018737.366389:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018737.366425:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333031040/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018737.366501:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018737.366506:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018737.366508:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018737.366513:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2999 00000100:00080000:3.0:1734018737.366517:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018737.366521:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018737.366532:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018737.366546:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333031168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018737.366556:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018737.366560:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018737.366561:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018737.366565:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2999 00000100:00080000:3.0:1734018737.366567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018737.366571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018737.366576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018737.366584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333031296/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018737.366594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018737.366597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018737.366599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018737.366603:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333031040:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734018737.366603:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 2999 00000100:00080000:3.0:1734018737.366605:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018737.366608:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018737.366612:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018737.366620:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333031424/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018737.366628:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018737.366631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018737.366633:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018737.366636:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2999 00000100:00080000:3.0:1734018737.366639:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018737.366642:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018737.366646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018737.366653:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333031552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018737.366662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018737.366665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018737.366667:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734018737.366670:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734018737.366671:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2999 00000100:00080000:3.0:1734018737.366674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018737.366678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018737.366682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018737.366689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333031680/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018737.366692:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018737/real 1734018737] req@ffff8800a73cdc00 x1818247333031040/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018737.366697:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018737.366700:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018737.366702:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018737.366705:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 2999 00000100:00080000:3.0:1734018737.366707:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018737.366711:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018737.366712:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333031040/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734018737.366715:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018737.366723:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333031808/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018737.366724:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018737.366727:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018737.366732:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333031040:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018737.366739:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333031168:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018737.366757:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018737.366763:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018737/real 1734018737] req@ffff8800a73cfb80 x1818247333031168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018737.366774:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333031168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018737.366782:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018737.366786:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018737.366789:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333031168:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018737.366812:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333031296:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018737.366827:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018737.366833:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018737/real 1734018737] req@ffff8800a73cf800 x1818247333031296/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018737.366844:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333031296/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018737.366851:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018737.366854:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018737.366857:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333031296:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018737.366862:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333031424:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018737.366874:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018737.366880:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018737/real 1734018737] req@ffff8800a73cdf80 x1818247333031424/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018737.366890:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333031424/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018737.366897:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018737.366899:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018737.366902:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333031424:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018737.366908:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333031552:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018737.366925:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018737.366931:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018737/real 1734018737] req@ffff8800a73cc700 x1818247333031552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018737.366941:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333031552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018737.366949:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018737.366951:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018737.366954:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333031552:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018737.366960:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333031680:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018737.366971:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018737.366977:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018737/real 1734018737] req@ffff8800a73cf100 x1818247333031680/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018737.366987:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333031680/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018737.366994:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018737.366996:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018737.366999:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333031680:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018737.367005:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333031808:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018737.367017:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018737.367022:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018737/real 1734018737] req@ffff8800a73cc380 x1818247333031808/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018737.367039:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333031808/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018737.367046:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018737.367049:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018737.367051:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333031808:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018738.180507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018739.182506:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018740.184482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018741.186542:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018742.188448:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018742.374396:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018742.374406:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018742.374412:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018742.374426:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3004 00000100:00080000:3.0:1734018742.374437:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018742.374446:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018742.374494:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333031936/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018742.374531:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018742.374536:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018742.374538:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018742.374542:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3004 00000100:00080000:3.0:1734018742.374546:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018742.374550:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018742.374562:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018742.374574:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333032064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018742.374584:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018742.374588:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018742.374590:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018742.374594:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3004 00000100:00080000:3.0:1734018742.374597:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018742.374601:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018742.374606:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018742.374615:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333032192/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018742.374623:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018742.374626:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018742.374628:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018742.374632:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3004 00000100:00080000:3.0:1734018742.374635:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018742.374638:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018742.374643:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018742.374652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333032320/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018742.374656:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333031936:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734018742.374662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018742.374666:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018742.374667:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018742.374671:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3004 00000100:00080000:3.0:1734018742.374674:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018742.374677:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018742.374682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018742.374690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333032448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018742.374699:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018742.374702:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018742.374703:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018742.374708:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3004 00000100:00080000:3.0:1734018742.374711:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018742.374715:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018742.374720:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018742.374728:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333032576/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018742.374733:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734018742.374737:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018742.374740:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018742.374742:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018742.374746:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3004 00000100:00080000:3.0:1734018742.374749:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018742.374752:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018742.374757:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018742.374759:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018742/real 1734018742] req@ffff8800a73cf480 x1818247333031936/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018742.374768:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333032704/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018742.374780:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333031936/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018742.374792:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018742.374796:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018742.374801:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333031936:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018742.374809:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333032064:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018742.374828:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018742.374835:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018742/real 1734018742] req@ffff8800a73cd500 x1818247333032064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018742.374846:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333032064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018742.374854:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018742.374858:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018742.374861:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333032064:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018742.374868:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333032192:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018742.374882:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018742.374888:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018742/real 1734018742] req@ffff8800a73cca80 x1818247333032192/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018742.374898:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333032192/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018742.374906:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018742.374909:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018742.374912:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333032192:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018742.374940:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333032320:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018742.374952:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018742.374958:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018742/real 1734018742] req@ffff8800a73cea00 x1818247333032320/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018742.374969:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333032320/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018742.374977:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018742.374980:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018742.374983:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333032320:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018742.374989:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333032448:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018742.375004:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018742.375010:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018742/real 1734018742] req@ffff8800a73cd880 x1818247333032448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018742.375020:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333032448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018742.375028:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018742.375030:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018742.375033:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333032448:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018742.375040:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333032576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018742.375052:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018742.375058:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018742/real 1734018742] req@ffff8800a73ced80 x1818247333032576/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018742.375068:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333032576/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018742.375075:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018742.375078:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018742.375081:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333032576:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018742.375087:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333032704:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018742.375099:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018742.375105:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018742/real 1734018742] req@ffff8800a73cce00 x1818247333032704/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018742.375121:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333032704/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018742.375128:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018742.375131:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018742.375134:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333032704:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018743.190526:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018744.192489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018745.194517:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018746.196557:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018747.198517:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018747.382367:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018747.382375:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018747.382380:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018747.382390:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3009 00000100:00080000:3.0:1734018747.382399:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018747.382407:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018747.382441:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333032832/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018747.382474:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018747.382478:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018747.382481:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018747.382486:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3009 00000100:00080000:3.0:1734018747.382489:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018747.382493:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018747.382503:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018747.382515:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333032960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018747.382525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018747.382528:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018747.382530:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018747.382534:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3009 00000100:00080000:3.0:1734018747.382536:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018747.382540:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018747.382544:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018747.382553:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333033088/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018747.382561:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018747.382564:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018747.382565:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018747.382569:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3009 00000100:00080000:3.0:1734018747.382571:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018747.382575:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018747.382578:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333032832:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734018747.382579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018747.382586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333033216/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018747.382596:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018747.382599:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018747.382601:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018747.382604:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3009 00000100:00080000:3.0:1734018747.382606:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018747.382609:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018747.382614:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018747.382621:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333033344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018747.382629:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018747.382632:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018747.382633:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018747.382637:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3009 00000100:00080000:3.0:1734018747.382640:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018747.382643:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734018747.382643:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018747.382647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018747.382654:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333033472/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018747.382662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018747.382665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734018747.382666:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018747/real 1734018747] req@ffff8800a73ce680 x1818247333032832/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018747.382667:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018747.382670:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3009 00000100:00080000:3.0:1734018747.382674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018747.382677:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018747.382682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018747.382685:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333032832/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018747.382690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333033600/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018747.382697:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018747.382700:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018747.382705:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333032832:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018747.382712:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333032960:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018747.382729:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018747.382736:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018747/real 1734018747] req@ffff8800a73cce00 x1818247333032960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018747.382747:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333032960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018747.382754:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018747.382757:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018747.382761:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333032960:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018747.382767:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333033088:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018747.382782:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018747.382788:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018747/real 1734018747] req@ffff8800a73ced80 x1818247333033088/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018747.382798:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333033088/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018747.382805:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018747.382808:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018747.382811:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333033088:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018747.382835:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333033216:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018747.382848:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018747.382854:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018747/real 1734018747] req@ffff8800a73cd880 x1818247333033216/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018747.382864:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333033216/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018747.382872:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018747.382874:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018747.382877:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333033216:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018747.382882:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333033344:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018747.382897:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018747.382903:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018747/real 1734018747] req@ffff8800a73cea00 x1818247333033344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018747.382913:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333033344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018747.382921:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018747.382923:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018747.382925:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333033344:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018747.382930:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333033472:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018747.382942:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018747.382947:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018747/real 1734018747] req@ffff8800a73cca80 x1818247333033472/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018747.382957:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333033472/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018747.382965:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018747.382967:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018747.382970:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333033472:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018747.382975:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333033600:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018747.382987:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018747.382992:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018747/real 1734018747] req@ffff8800a73cd500 x1818247333033600/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018747.383009:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333033600/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018747.383016:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018747.383018:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018747.383021:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333033600:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018748.200449:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018749.202508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018750.204478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018751.206488:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018752.208493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018752.390331:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018752.390339:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018752.390344:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018752.390354:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3014 00000100:00080000:3.0:1734018752.390363:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018752.390371:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018752.390406:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333033728/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018752.390440:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018752.390444:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018752.390446:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018752.390451:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3014 00000100:00080000:3.0:1734018752.390454:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018752.390458:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018752.390468:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018752.390479:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333033856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018752.390489:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018752.390492:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018752.390494:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018752.390498:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3014 00000100:00080000:3.0:1734018752.390501:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018752.390504:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018752.390509:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018752.390517:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333033984/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018752.390525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018752.390528:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018752.390530:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018752.390533:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3014 00000100:00080000:3.0:1734018752.390536:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018752.390539:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018752.390543:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018752.390544:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333033728:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734018752.390551:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333034112/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018752.390559:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018752.390563:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018752.390564:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018752.390568:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3014 00000100:00080000:3.0:1734018752.390570:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018752.390574:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018752.390578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018752.390586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333034240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018752.390594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018752.390597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018752.390599:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018752.390602:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3014 00000100:00080000:3.0:1734018752.390605:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018752.390609:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018752.390610:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734018752.390613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018752.390620:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333034368/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018752.390628:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018752.390631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734018752.390632:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018752/real 1734018752] req@ffff8800a73cf480 x1818247333033728/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018752.390632:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018752.390636:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3014 00000100:00080000:3.0:1734018752.390639:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018752.390642:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018752.390646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018752.390650:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333033728/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734018752.390655:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333034496/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018752.390661:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018752.390665:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018752.390670:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333033728:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018752.390677:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333033856:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018752.390694:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018752.390701:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018752/real 1734018752] req@ffff8800a73cc380 x1818247333033856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018752.390712:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333033856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018752.390720:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018752.390723:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018752.390727:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333033856:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018752.390733:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333033984:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018752.390747:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018752.390753:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018752/real 1734018752] req@ffff8800a73cf100 x1818247333033984/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018752.390763:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333033984/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018752.390771:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018752.390773:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018752.390776:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333033984:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018752.390800:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333034112:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018752.390817:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018752.390824:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018752/real 1734018752] req@ffff8800a73cc700 x1818247333034112/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018752.390834:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333034112/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018752.390841:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018752.390843:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018752.390846:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333034112:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018752.390851:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333034240:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018752.390866:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018752.390873:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018752/real 1734018752] req@ffff8800a73cdf80 x1818247333034240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018752.390883:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333034240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018752.390890:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018752.390892:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018752.390895:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333034240:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018752.390900:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333034368:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018752.390911:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018752.390917:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018752/real 1734018752] req@ffff8800a73cf800 x1818247333034368/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018752.390927:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333034368/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018752.390934:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018752.390936:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018752.390940:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333034368:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018752.390945:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333034496:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018752.390957:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018752.390962:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018752/real 1734018752] req@ffff8800a73cfb80 x1818247333034496/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018752.390979:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333034496/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018752.390986:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018752.390988:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018752.390991:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333034496:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018753.210467:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018754.212441:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018755.214466:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018756.216456:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018757.218507:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734018757.398365:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018757.398374:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734018757.398378:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018757.398389:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3019 00000100:00080000:3.0:1734018757.398398:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018757.398406:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734018757.398442:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333034624/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018757.398475:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018757.398479:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018757.398481:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018757.398485:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3019 00000100:00080000:3.0:1734018757.398489:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018757.398492:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018757.398507:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018757.398519:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333034752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018757.398529:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018757.398532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018757.398535:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018757.398539:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3019 00000100:00080000:3.0:1734018757.398541:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018757.398545:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018757.398549:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734018757.398558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333034880/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018757.398567:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018757.398570:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018757.398572:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018757.398575:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3019 00000100:00100000:1.0:1734018757.398578:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333034624:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734018757.398578:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018757.398581:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018757.398586:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018757.398594:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333035008/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018757.398602:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018757.398605:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734018757.398606:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018757.398610:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3019 00000100:00080000:3.0:1734018757.398612:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018757.398615:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018757.398620:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734018757.398627:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333035136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734018757.398636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018757.398639:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734018757.398641:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734018757.398645:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734018757.398645:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3019 00000100:00080000:3.0:1734018757.398648:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018757.398652:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734018757.398656:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734018757.398663:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333035264/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018757.398667:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018757/real 1734018757] req@ffff8800a73cdc00 x1818247333034624/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734018757.398671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734018757.398674:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734018757.398676:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734018757.398679:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3019 00000100:00080000:3.0:1734018757.398682:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734018757.398685:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018757.398687:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333034624/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734018757.398689:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734018757.398697:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1734018757.398697:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333035392/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018757.398701:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018757.398705:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333034624:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018757.398713:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333034752:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018757.398730:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018757.398737:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018757/real 1734018757] req@ffff8800a73cfb80 x1818247333034752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018757.398747:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333034752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018757.398755:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018757.398759:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018757.398762:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333034752:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018757.398784:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333034880:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018757.398798:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018757.398805:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018757/real 1734018757] req@ffff8800a73cf800 x1818247333034880/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018757.398815:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333034880/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018757.398823:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018757.398825:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018757.398828:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333034880:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018757.398833:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333035008:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018757.398845:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018757.398851:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018757/real 1734018757] req@ffff8800a73cdf80 x1818247333035008/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018757.398860:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333035008/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018757.398868:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018757.398870:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018757.398873:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333035008:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018757.398879:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333035136:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018757.398897:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018757.398903:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018757/real 1734018757] req@ffff8800a73cc700 x1818247333035136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018757.398914:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333035136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018757.398921:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018757.398923:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018757.398926:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333035136:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018757.398931:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333035264:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018757.398943:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018757.398949:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018757/real 1734018757] req@ffff8800a73cf100 x1818247333035264/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018757.398959:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333035264/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018757.398966:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018757.398969:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018757.398972:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333035264:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018757.398977:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333035392:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018757.398988:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018757.398994:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018757/real 1734018757] req@ffff8800a73cc380 x1818247333035392/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018757.399010:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333035392/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018757.399018:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018757.399020:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018757.399023:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333035392:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018758.220549:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018759.222482:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018760.224516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018761.226491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018762.228419:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018762.406464:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018762.406472:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018762.406476:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018762.406486:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3024 00000100:00080000:2.0:1734018762.406495:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018762.406503:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018762.406539:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333035520/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018762.406579:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018762.406583:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018762.406586:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018762.406591:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3024 00000100:00080000:2.0:1734018762.406594:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018762.406598:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018762.406607:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018762.406618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333035648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018762.406634:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018762.406638:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018762.406640:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018762.406644:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3024 00000100:00080000:2.0:1734018762.406647:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018762.406650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018762.406655:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018762.406663:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333035776/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018762.406672:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018762.406675:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018762.406676:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018762.406680:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3024 00000100:00080000:2.0:1734018762.406682:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018762.406686:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018762.406690:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018762.406694:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333035520:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734018762.406699:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247333035904/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018762.406709:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018762.406712:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018762.406714:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018762.406717:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3024 00000100:00080000:2.0:1734018762.406720:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018762.406723:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018762.406727:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018762.406735:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247333036032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018762.406743:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018762.406746:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018762.406748:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018762.406751:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3024 00000100:00080000:2.0:1734018762.406754:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018762.406758:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018762.406760:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:2.0:1734018762.406763:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018762.406770:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247333036160/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018762.406778:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018762.406781:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734018762.406782:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018762/real 1734018762] req@ffff8800a73cad80 x1818247333035520/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018762.406783:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018762.406786:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3024 00000100:00080000:2.0:1734018762.406790:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018762.406793:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018762.406797:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018762.406802:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333035520/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018762.406805:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333036288/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018762.406812:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018762.406815:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018762.406820:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333035520:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018762.406827:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333035648:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018762.406844:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018762.406851:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018762/real 1734018762] req@ffff8800a73cb100 x1818247333035648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018762.406862:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333035648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018762.406869:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018762.406873:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018762.406876:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333035648:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018762.406882:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333035776:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018762.406896:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018762.406902:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018762/real 1734018762] req@ffff8800a73c9880 x1818247333035776/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018762.406912:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333035776/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018762.406920:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018762.406922:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018762.406925:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333035776:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018762.406953:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333035904:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018762.406965:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018762.406971:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018762/real 1734018762] req@ffff8800a73c9c00 x1818247333035904/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018762.406982:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247333035904/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018762.406989:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018762.406991:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018762.406994:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333035904:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018762.407000:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333036032:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018762.407015:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018762.407074:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018762/real 1734018762] req@ffff8800a73c9500 x1818247333036032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018762.407087:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247333036032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018762.407095:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018762.407098:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018762.407101:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333036032:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018762.407107:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333036160:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018762.407123:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018762.407130:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018762/real 1734018762] req@ffff8800a73c8000 x1818247333036160/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018762.407140:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247333036160/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018762.407148:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018762.407150:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018762.407153:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333036160:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018762.407158:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333036288:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018762.407170:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018762.407175:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018762/real 1734018762] req@ffff8800a73c8e00 x1818247333036288/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018762.407192:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333036288/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018762.407199:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018762.407201:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018762.407204:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333036288:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018763.230420:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018764.232497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018765.234464:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018766.236497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018767.238512:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018767.414364:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018767.414372:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018767.414377:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018767.414387:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3029 00000100:00080000:2.0:1734018767.414397:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018767.414404:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018767.414440:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247333036416/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018767.414473:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018767.414477:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018767.414479:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018767.414484:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3029 00000100:00080000:2.0:1734018767.414487:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018767.414491:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018767.414505:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018767.414516:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247333036544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018767.414526:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018767.414529:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018767.414532:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018767.414536:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3029 00000100:00080000:2.0:1734018767.414538:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018767.414542:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018767.414547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018767.414556:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247333036672/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018767.414566:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018767.414569:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018767.414570:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018767.414574:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3029 00000100:00080000:2.0:1734018767.414577:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018767.414578:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333036416:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734018767.414580:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018767.414584:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018767.414593:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333036800/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018767.414601:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018767.414604:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018767.414605:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018767.414609:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3029 00000100:00080000:2.0:1734018767.414611:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018767.414615:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018767.414619:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018767.414626:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247333036928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018767.414636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018767.414639:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018767.414640:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018767.414644:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3029 00000800:00000100:1.0:1734018767.414645:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:2.0:1734018767.414647:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018767.414650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018767.414654:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018767.414661:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247333037056/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018767.414667:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018767/real 1734018767] req@ffff8800a73caa00 x1818247333036416/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018767.414669:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018767.414672:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018767.414674:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018767.414677:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3029 00000100:00080000:2.0:1734018767.414680:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018767.414683:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018767.414687:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73caa00 x1818247333036416/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734018767.414687:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018767.414694:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247333037184/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018767.414697:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018767.414701:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018767.414705:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333036416:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018767.414713:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333036544:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018767.414730:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018767.414737:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018767/real 1734018767] req@ffff8800a73c9f80 x1818247333036544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018767.414748:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9f80 x1818247333036544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018767.414755:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018767.414759:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018767.414762:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333036544:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018767.414784:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333036672:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018767.414799:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018767.414805:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018767/real 1734018767] req@ffff8800a73ca680 x1818247333036672/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018767.414816:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ca680 x1818247333036672/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018767.414824:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018767.414826:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018767.414829:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333036672:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018767.414835:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333036800:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018767.414846:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018767.414852:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018767/real 1734018767] req@ffff8800a73c8e00 x1818247333036800/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018767.414862:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333036800/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018767.414869:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018767.414871:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018767.414874:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333036800:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018767.414880:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333036928:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018767.414896:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018767.414902:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018767/real 1734018767] req@ffff8800a73c8000 x1818247333036928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018767.414913:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247333036928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018767.414920:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018767.414922:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018767.414925:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333036928:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018767.414931:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333037056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018767.414993:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018767.415001:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018767/real 1734018767] req@ffff8800a73c9500 x1818247333037056/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018767.415011:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247333037056/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018767.415019:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018767.415021:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018767.415024:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333037056:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018767.415029:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333037184:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018767.415042:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018767.415049:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018767/real 1734018767] req@ffff8800a73c9c00 x1818247333037184/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018767.415065:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247333037184/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018767.415073:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018767.415075:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018767.415078:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333037184:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018768.240502:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018769.242501:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018770.244503:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018771.246505:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018772.248447:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018772.422365:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018772.422377:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018772.422383:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018772.422396:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3034 00000100:00080000:2.0:1734018772.422408:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018772.422419:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018772.422467:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333037312/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018772.422506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018772.422511:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018772.422513:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018772.422518:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3034 00000100:00080000:2.0:1734018772.422521:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018772.422526:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018772.422538:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018772.422550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333037440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018772.422561:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018772.422565:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018772.422566:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018772.422571:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3034 00000100:00080000:2.0:1734018772.422573:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018772.422577:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018772.422583:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018772.422591:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333037568/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018772.422600:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018772.422603:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018772.422605:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018772.422609:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3034 00000100:00080000:2.0:1734018772.422612:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018772.422616:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018772.422621:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018772.422626:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333037312:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734018772.422632:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247333037696/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018772.422642:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018772.422645:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018772.422647:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018772.422651:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3034 00000100:00080000:2.0:1734018772.422654:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018772.422658:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018772.422662:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018772.422672:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247333037824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018772.422681:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018772.422685:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018772.422686:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018772.422691:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3034 00000100:00080000:2.0:1734018772.422694:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018772.422698:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018772.422702:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018772.422710:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247333037952/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018772.422720:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018772.422723:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018772.422725:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018772.422727:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333037440:192.168.201.152@tcp:38:kworker.0 00000100:00080000:2.0:1734018772.422729:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3034 00000100:00080000:2.0:1734018772.422731:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018772.422735:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018772.422739:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018772.422747:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333038080/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018772.422750:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333037568:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018772.422776:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333037696:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018772.422792:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333037824:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018772.422809:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333037952:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018772.422823:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333038080:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018773.250509:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018774.252508:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018775.254494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734018775.428515:0:1955:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:3.0:1734018775.428564:0:1955:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734018775.428586:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734018775.428621:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734018775.428629:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734018775.428636:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734018775.428642:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734018775.428649:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734018775.428656:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734018775.428737:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018772/real 1734018775] req@ffff8800a73c9c00 x1818247333037696/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018775.428767:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247333037696/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018775.428779:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018775.428784:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018775.428789:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333037696:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018775.428797:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018772/real 1734018775] req@ffff8800a73c9500 x1818247333037824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018775.428862:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247333037824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018775.428870:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018775.428873:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018775.428877:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333037824:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018775.428884:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018772/real 1734018775] req@ffff8800a73c8000 x1818247333037952/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018775.428895:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247333037952/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018775.428902:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018775.428905:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018775.428908:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333037952:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018775.428914:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018772/real 1734018775] req@ffff8800a73c8e00 x1818247333038080/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018775.428925:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333038080/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018775.428932:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018775.428935:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018775.428938:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333038080:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018775.428947:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018772/real 1734018775] req@ffff8800a73c9880 x1818247333037312/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018775.428964:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333037312/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018775.428977:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018775.428980:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018775.428985:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333037312:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018775.428995:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018772/real 1734018775] req@ffff8800a73cb100 x1818247333037440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018775.429012:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333037440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018775.429024:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018775.429028:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018775.429033:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333037440:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018775.429042:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018772/real 1734018775] req@ffff8800a73cad80 x1818247333037568/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018775.429060:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333037568/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018775.429072:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018775.429076:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018775.429082:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333037568:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018776.256491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018777.258487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018777.430341:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018777.430350:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018777.430355:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018777.430365:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3039 00000100:00080000:2.0:1734018777.430376:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018777.430384:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018777.430420:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247333038208/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018777.430455:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018777.430459:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018777.430461:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018777.430466:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3039 00000100:00080000:2.0:1734018777.430469:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018777.430473:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018777.430484:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018777.430495:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247333038336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018777.430506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018777.430509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018777.430511:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018777.430515:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3039 00000100:00080000:2.0:1734018777.430518:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018777.430522:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018777.430526:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018777.430535:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247333038464/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018777.430543:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018777.430546:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018777.430548:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018777.430551:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3039 00000100:00080000:2.0:1734018777.430554:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018777.430557:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018777.430562:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018777.430563:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333038208:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734018777.430570:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333038592/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018777.430579:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018777.430582:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018777.430584:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018777.430588:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3039 00000100:00080000:2.0:1734018777.430591:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018777.430595:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018777.430599:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018777.430607:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333038720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018777.430615:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018777.430618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018777.430619:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018777.430623:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3039 00000100:00080000:2.0:1734018777.430626:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018777.430629:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018777.430630:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:2.0:1734018777.430633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018777.430640:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333038848/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018777.430648:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018777.430651:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018777/real 1734018777] req@ffff8800a73ca680 x1818247333038208/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018777.430652:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018777.430653:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018777.430656:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3039 00000100:00080000:2.0:1734018777.430659:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018777.430662:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018777.430667:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018777.430672:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ca680 x1818247333038208/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018777.430675:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333038976/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018777.430683:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018777.430686:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018777.430691:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333038208:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018777.430698:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333038336:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018777.430716:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018777.430722:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018777/real 1734018777] req@ffff8800a73c9f80 x1818247333038336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018777.430733:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9f80 x1818247333038336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018777.430741:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018777.430744:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018777.430747:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333038336:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018777.430753:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333038464:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018777.430813:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018777.430822:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018777/real 1734018777] req@ffff8800a73caa00 x1818247333038464/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018777.430833:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73caa00 x1818247333038464/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018777.430842:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018777.430845:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018777.430848:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333038464:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018777.430873:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333038592:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018777.430890:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018777.430896:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018777/real 1734018777] req@ffff8800a73cad80 x1818247333038592/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018777.430906:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333038592/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018777.430914:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018777.430916:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018777.430919:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333038592:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018777.430925:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333038720:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018777.430937:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018777.430943:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018777/real 1734018777] req@ffff8800a73cb100 x1818247333038720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018777.430952:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333038720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018777.430960:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018777.430962:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018777.430965:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333038720:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018777.430971:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333038848:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018777.430983:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018777.430989:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018777/real 1734018777] req@ffff8800a73c9880 x1818247333038848/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018777.430999:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333038848/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018777.431007:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018777.431009:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018777.431012:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333038848:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018777.431017:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333038976:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018777.431028:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018777.431034:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018777/real 1734018777] req@ffff8800a73c8e00 x1818247333038976/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018777.431044:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333038976/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018777.431051:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018777.431053:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018777.431056:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333038976:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018778.260497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018779.262519:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018780.264502:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018781.266510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018782.268497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018782.438331:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018782.438341:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018782.438347:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018782.438361:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3044 00000100:00080000:2.0:1734018782.438372:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018782.438382:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018782.438430:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247333039104/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018782.438469:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018782.438474:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018782.438476:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018782.438481:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3044 00000100:00080000:2.0:1734018782.438484:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018782.438489:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018782.438501:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018782.438512:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247333039232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018782.438522:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018782.438526:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018782.438528:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018782.438532:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3044 00000100:00080000:2.0:1734018782.438535:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018782.438538:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018782.438545:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018782.438554:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247333039360/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018782.438564:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018782.438567:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018782.438569:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018782.438572:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3044 00000100:00080000:2.0:1734018782.438575:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018782.438578:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018782.438583:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018782.438594:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333039488/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018782.438595:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333039104:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734018782.438604:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018782.438607:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018782.438609:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018782.438613:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3044 00000100:00080000:2.0:1734018782.438615:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018782.438619:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018782.438624:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018782.438632:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333039616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018782.438642:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018782.438646:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018782.438648:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018782.438653:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3044 00000100:00080000:2.0:1734018782.438666:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018782.438670:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018782.438671:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:2.0:1734018782.438676:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018782.438719:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333039744/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018782.438730:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018782.438734:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018782.438735:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018782.438736:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018782/real 1734018782] req@ffff8800a73c8000 x1818247333039104/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018782.438739:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3044 00000100:00080000:2.0:1734018782.438742:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018782.438746:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018782.438750:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018782.438756:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247333039104/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018782.438759:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333039872/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018782.438769:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018782.438773:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018782.438778:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333039104:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018782.438787:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333039232:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018782.438809:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018782.438817:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018782/real 1734018782] req@ffff8800a73c9500 x1818247333039232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018782.438828:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247333039232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018782.438836:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018782.438839:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018782.438843:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333039232:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018782.438849:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333039360:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018782.438863:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018782.438869:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018782/real 1734018782] req@ffff8800a73c9c00 x1818247333039360/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018782.438879:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247333039360/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018782.438887:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018782.438890:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018782.438893:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333039360:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018782.438919:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333039488:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018782.438932:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018782.438938:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018782/real 1734018782] req@ffff8800a73c8e00 x1818247333039488/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018782.438948:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333039488/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018782.438956:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018782.438958:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018782.438961:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333039488:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018782.438967:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333039616:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018782.438979:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018782.438985:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018782/real 1734018782] req@ffff8800a73c9880 x1818247333039616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018782.438995:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333039616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018782.439002:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018782.439004:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018782.439008:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333039616:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018782.439014:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333039744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018782.439030:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018782.439036:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018782/real 1734018782] req@ffff8800a73cb100 x1818247333039744/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018782.439046:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333039744/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018782.439053:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018782.439055:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018782.439058:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333039744:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018782.439064:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333039872:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018782.439076:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018782.439082:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018782/real 1734018782] req@ffff8800a73cad80 x1818247333039872/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018782.439092:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333039872/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018782.439100:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018782.439102:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018782.439105:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333039872:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018783.270518:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018784.272469:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018785.274484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018786.276493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018787.278498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018787.446337:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018787.446346:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018787.446350:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018787.446360:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3049 00000100:00080000:2.0:1734018787.446369:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018787.446377:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018787.446413:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247333040000/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018787.446456:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018787.446462:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018787.446465:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018787.446472:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3049 00000100:00080000:2.0:1734018787.446477:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018787.446483:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018787.446496:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018787.446515:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247333040128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018787.446522:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333040000:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734018787.446531:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018787.446535:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018787.446537:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018787.446542:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3049 00000100:00080000:2.0:1734018787.446546:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018787.446551:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018787.446557:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018787.446570:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247333040256/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018787.446583:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018787.446587:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018787.446589:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734018787.446591:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:2.0:1734018787.446594:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3049 00000100:00080000:2.0:1734018787.446644:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018787.446650:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018787/real 1734018787] req@ffff8800a73caa00 x1818247333040000/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018787.446651:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018787.446659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018787.446674:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333040384/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018787.446684:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73caa00 x1818247333040000/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018787.446688:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018787.446694:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018787.446696:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734018787.446697:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018787.446702:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:2.0:1734018787.446702:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3049 00000100:00080000:2.0:1734018787.446706:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018787.446708:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333040000:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734018787.446711:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018787.446718:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018787.446729:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333040512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018787.446736:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333040128:192.168.201.152@tcp:38:kworker.0 00000100:00080000:2.0:1734018787.446743:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018787.446747:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018787.446750:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018787.446755:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3049 00000100:00080000:2.0:1734018787.446761:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018787.446766:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:2.0:1734018787.446766:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018787.446773:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018787.446777:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018787/real 1734018787] req@ffff8800a73c9f80 x1818247333040128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018787.446785:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333040640/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018787.446795:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9f80 x1818247333040128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018787.446799:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018787.446805:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1734018787.446807:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018787.446808:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734018787.446811:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00080000:2.0:1734018787.446813:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3049 00000100:00100000:1.0:1734018787.446816:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333040128:192.168.201.152@tcp:38:kworker.0 00000100:00080000:2.0:1734018787.446818:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018787.446823:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018787.446825:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333040256:192.168.201.152@tcp:8:kworker.0 00000008:00100000:2.0:1734018787.446829:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018787.446841:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333040768/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018787.446848:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018787.446858:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018787/real 1734018787] req@ffff8800a73ca680 x1818247333040256/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018787.446875:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ca680 x1818247333040256/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018787.446886:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018787.446890:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018787.446895:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333040256:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018787.446904:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333040384:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018787.446928:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018787.446939:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018787/real 1734018787] req@ffff8800a73cad80 x1818247333040384/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018787.446957:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333040384/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018787.446965:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018787.446967:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018787.446971:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333040384:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018787.446988:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333040512:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018787.447005:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018787.447011:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018787/real 1734018787] req@ffff8800a73cb100 x1818247333040512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018787.447022:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333040512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018787.447030:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018787.447032:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018787.447035:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333040512:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018787.447040:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333040640:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018787.447055:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018787.447061:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018787/real 1734018787] req@ffff8800a73c9880 x1818247333040640/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018787.447071:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333040640/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018787.447078:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018787.447080:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018787.447083:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333040640:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018787.447088:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333040768:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018787.447099:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018787.447104:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018787/real 1734018787] req@ffff8800a73c8e00 x1818247333040768/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018787.447114:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333040768/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018787.447121:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018787.447123:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018787.447126:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333040768:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018788.280471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018789.282516:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018790.284501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018791.286485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018792.288482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018792.454354:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018792.454364:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018792.454368:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018792.454378:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3054 00000100:00080000:2.0:1734018792.454388:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018792.454396:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018792.454432:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247333040896/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018792.454465:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018792.454470:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018792.454472:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018792.454476:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3054 00000100:00080000:2.0:1734018792.454480:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018792.454484:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018792.454493:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018792.454504:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247333041024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018792.454552:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018792.454555:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018792.454557:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018792.454561:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3054 00000100:00080000:2.0:1734018792.454564:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018792.454567:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018792.454573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018792.454582:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247333041152/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018792.454591:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018792.454594:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734018792.454596:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333040896:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734018792.454596:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018792.454600:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3054 00000100:00080000:2.0:1734018792.454602:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018792.454606:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018792.454610:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018792.454619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333041280/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018792.454627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018792.454630:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018792.454632:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018792.454635:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3054 00000100:00080000:2.0:1734018792.454638:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018792.454642:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018792.454646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018792.454654:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333041408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018792.454662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1734018792.454664:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:2.0:1734018792.454665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018792.454666:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018792.454670:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3054 00000100:00080000:2.0:1734018792.454673:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018792.454676:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018792.454680:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018792.454686:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018792/real 1734018792] req@ffff8800a73c9c00 x1818247333040896/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018792.454688:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333041536/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018792.454696:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018792.454699:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018792.454701:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018792.454704:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3054 00000100:00080000:2.0:1734018792.454706:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018792.454710:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018792.454713:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247333040896/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734018792.454714:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018792.454721:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333041664/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018792.454723:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018792.454726:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018792.454731:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333040896:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018792.454738:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333041024:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018792.454756:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018792.454763:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018792/real 1734018792] req@ffff8800a73c9500 x1818247333041024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018792.454774:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247333041024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018792.454781:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018792.454784:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018792.454787:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333041024:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018792.454809:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333041152:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018792.454822:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018792.454828:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018792/real 1734018792] req@ffff8800a73c8000 x1818247333041152/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018792.454838:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247333041152/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018792.454846:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018792.454850:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018792.454853:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333041152:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018792.454859:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333041280:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018792.454871:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018792.454877:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018792/real 1734018792] req@ffff8800a73c8e00 x1818247333041280/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018792.454887:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333041280/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018792.454894:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018792.454897:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018792.454900:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333041280:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018792.454905:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333041408:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018792.454917:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018792.454923:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018792/real 1734018792] req@ffff8800a73c9880 x1818247333041408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018792.454933:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333041408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018792.454940:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018792.454942:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018792.454945:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333041408:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018792.454951:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333041536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018792.454967:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018792.454973:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018792/real 1734018792] req@ffff8800a73cb100 x1818247333041536/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018792.454983:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333041536/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018792.454990:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018792.454992:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018792.454995:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333041536:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018792.455001:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333041664:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018792.455012:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018792.455018:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018792/real 1734018792] req@ffff8800a73cad80 x1818247333041664/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018792.455028:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333041664/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018792.455036:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018792.455038:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018792.455041:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333041664:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018793.290537:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018794.292549:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018795.294515:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018796.296537:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018797.298500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018797.462330:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018797.462339:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018797.462345:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018797.462358:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3060 00000100:00080000:2.0:1734018797.462369:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018797.462379:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018797.462458:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247333041792/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018797.462494:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018797.462499:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018797.462501:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018797.462507:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3060 00000100:00080000:2.0:1734018797.462510:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018797.462514:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018797.462527:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018797.462540:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247333041920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018797.462551:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018797.462555:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018797.462557:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018797.462561:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3060 00000100:00080000:2.0:1734018797.462564:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018797.462568:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018797.462575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018797.462585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247333042048/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018797.462596:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018797.462599:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018797.462601:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018797.462605:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3060 00000100:00080000:2.0:1734018797.462607:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018797.462611:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018797.462612:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333041792:192.168.201.152@tcp:250:kworker.0 00000008:00100000:2.0:1734018797.462616:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018797.462626:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333042176/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018797.462634:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018797.462638:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018797.462639:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018797.462643:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3060 00000100:00080000:2.0:1734018797.462646:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018797.462649:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018797.462654:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018797.462663:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333042304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018797.462671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018797.462674:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018797.462677:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018797.462682:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3060 00000100:00080000:2.0:1734018797.462685:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734018797.462689:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:2.0:1734018797.462689:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018797.462694:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018797.462702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333042432/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018797.462710:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018797.462714:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018797/real 1734018797] req@ffff8800a73ca680 x1818247333041792/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018797.462714:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018797.462715:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018797.462719:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3060 00000100:00080000:2.0:1734018797.462722:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018797.462726:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018797.462730:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018797.462738:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333042560/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018797.462742:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ca680 x1818247333041792/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018797.462754:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018797.462758:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018797.462763:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333041792:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018797.462771:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333041920:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018797.462790:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018797.462797:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018797/real 1734018797] req@ffff8800a73c9f80 x1818247333041920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018797.462808:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9f80 x1818247333041920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018797.462816:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018797.462819:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018797.462822:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333041920:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018797.462848:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333042048:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018797.462863:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018797.462869:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018797/real 1734018797] req@ffff8800a73caa00 x1818247333042048/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018797.462880:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73caa00 x1818247333042048/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018797.462888:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018797.462891:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018797.462894:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333042048:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018797.462900:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333042176:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018797.462913:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018797.462920:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018797/real 1734018797] req@ffff8800a73cad80 x1818247333042176/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018797.462929:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333042176/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018797.462937:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018797.462940:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018797.462943:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333042176:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018797.462949:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333042304:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018797.462961:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018797.462967:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018797/real 1734018797] req@ffff8800a73cb100 x1818247333042304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018797.462977:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333042304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018797.462984:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018797.462987:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018797.462990:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333042304:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018797.462996:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333042432:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018797.463012:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018797.463018:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018797/real 1734018797] req@ffff8800a73c9880 x1818247333042432/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018797.463028:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333042432/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018797.463035:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018797.463038:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018797.463040:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333042432:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018797.463046:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333042560:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018797.463058:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018797.463064:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018797/real 1734018797] req@ffff8800a73c8e00 x1818247333042560/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018797.463074:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333042560/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018797.463081:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018797.463083:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018797.463086:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333042560:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018798.300477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018799.302477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018800.304447:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018801.306550:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018802.308493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018802.470399:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018802.470409:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018802.470415:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018802.470428:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3065 00000100:00080000:2.0:1734018802.470439:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018802.470448:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018802.470497:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247333042688/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018802.470544:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018802.470549:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018802.470551:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018802.470556:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3065 00000100:00080000:2.0:1734018802.470559:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018802.470564:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018802.470576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018802.470588:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247333042816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018802.470599:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018802.470603:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018802.470604:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018802.470609:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3065 00000100:00080000:2.0:1734018802.470611:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018802.470612:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333042688:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734018802.470615:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018802.470620:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018802.470629:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247333042944/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018802.470638:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018802.470641:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018802.470643:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018802.470647:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3065 00000100:00080000:2.0:1734018802.470650:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018802.470654:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018802.470659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018802.470669:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333043072/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018802.470677:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018802.470680:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018802.470682:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018802.470685:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3065 00000800:00000100:1.0:1734018802.470689:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:2.0:1734018802.470689:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018802.470692:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018802.470697:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018802.470705:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333043200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018802.470713:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018802.470714:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018802/real 1734018802] req@ffff8800a73c8000 x1818247333042688/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018802.470716:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018802.470718:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018802.470723:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3065 00000100:00080000:2.0:1734018802.470726:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018802.470730:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018802.470735:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018802.470742:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247333042688/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018802.470743:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333043328/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018802.470752:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1734018802.470754:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018802.470756:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1734018802.470758:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:2.0:1734018802.470758:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018802.470762:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3065 00000100:00100000:1.0:1734018802.470763:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333042688:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734018802.470765:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018802.470769:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018802.470773:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018802.470781:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333043456/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018802.470789:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333042816:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018802.470808:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018802.470816:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018802/real 1734018802] req@ffff8800a73c9500 x1818247333042816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018802.470828:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247333042816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018802.470836:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018802.470839:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018802.470842:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333042816:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018802.470849:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333042944:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018802.470863:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018802.470870:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018802/real 1734018802] req@ffff8800a73c9c00 x1818247333042944/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018802.470880:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247333042944/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018802.470888:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018802.470891:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018802.470894:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333042944:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018802.470901:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333043072:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018802.470913:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018802.470919:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018802/real 1734018802] req@ffff8800a73c8e00 x1818247333043072/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018802.470929:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333043072/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018802.470937:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018802.470940:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018802.470943:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333043072:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018802.470948:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333043200:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018802.470961:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018802.470967:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018802/real 1734018802] req@ffff8800a73c9880 x1818247333043200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018802.470977:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333043200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018802.470984:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018802.470987:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018802.470990:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333043200:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018802.470996:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333043328:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018802.471012:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018802.471018:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018802/real 1734018802] req@ffff8800a73cb100 x1818247333043328/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018802.471028:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333043328/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018802.471035:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018802.471038:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018802.471041:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333043328:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018802.471059:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333043456:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018802.471071:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018802.471077:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018802/real 1734018802] req@ffff8800a73cad80 x1818247333043456/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018802.471087:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333043456/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018802.471094:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018802.471097:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018802.471100:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333043456:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018803.310517:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018804.312529:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018805.314493:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018806.316490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018807.318451:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018807.478359:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018807.478368:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018807.478372:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018807.478383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3070 00000100:00080000:2.0:1734018807.478393:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018807.478401:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018807.478438:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247333043584/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018807.478471:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018807.478475:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018807.478477:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018807.478482:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3070 00000100:00080000:2.0:1734018807.478485:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018807.478489:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018807.478499:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018807.478509:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247333043712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018807.478521:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018807.478524:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018807.478525:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018807.478529:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3070 00000100:00080000:2.0:1734018807.478532:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018807.478535:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018807.478540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018807.478549:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247333043840/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018807.478557:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018807.478560:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018807.478562:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018807.478565:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3070 00000100:00080000:2.0:1734018807.478567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018807.478571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018807.478575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018807.478580:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333043584:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734018807.478584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333043968/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018807.478593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018807.478596:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018807.478598:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018807.478601:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3070 00000100:00080000:2.0:1734018807.478604:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018807.478608:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018807.478612:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018807.478619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333044096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018807.478627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018807.478630:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018807.478632:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018807.478636:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3070 00000100:00080000:2.0:1734018807.478638:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018807.478642:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018807.478646:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:2.0:1734018807.478646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018807.478653:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333044224/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018807.478661:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018807.478664:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018807.478666:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018807.478668:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018807/real 1734018807] req@ffff8800a73caa00 x1818247333043584/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018807.478669:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3070 00000100:00080000:2.0:1734018807.478672:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018807.478675:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018807.478679:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018807.478687:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333044352/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018807.478695:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73caa00 x1818247333043584/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018807.478705:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018807.478708:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018807.478713:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333043584:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018807.478720:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333043712:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018807.478737:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018807.478743:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018807/real 1734018807] req@ffff8800a73c9f80 x1818247333043712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018807.478754:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9f80 x1818247333043712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018807.478762:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018807.478765:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018807.478768:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333043712:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018807.478774:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333043840:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018807.478787:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018807.478793:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018807/real 1734018807] req@ffff8800a73ca680 x1818247333043840/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018807.478803:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ca680 x1818247333043840/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018807.478811:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018807.478813:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018807.478816:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333043840:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018807.478842:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333043968:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018807.478855:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018807.478860:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018807/real 1734018807] req@ffff8800a73cad80 x1818247333043968/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018807.478871:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333043968/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018807.478879:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018807.478881:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018807.478884:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333043968:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018807.478889:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333044096:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018807.478901:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018807.478906:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018807/real 1734018807] req@ffff8800a73cb100 x1818247333044096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018807.478916:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333044096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018807.478923:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018807.478926:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018807.478929:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333044096:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018807.478935:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333044224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018807.478950:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018807.478955:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018807/real 1734018807] req@ffff8800a73c9880 x1818247333044224/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018807.478966:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333044224/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018807.478973:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018807.478975:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018807.478979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333044224:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018807.478984:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333044352:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018807.478996:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018807.479001:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018807/real 1734018807] req@ffff8800a73c8e00 x1818247333044352/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018807.479011:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333044352/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018807.479018:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018807.479020:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018807.479023:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333044352:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018808.320496:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018809.322541:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018810.324516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018811.326492:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018812.328454:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018812.486352:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018812.486361:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018812.486366:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018812.486376:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3075 00000100:00080000:2.0:1734018812.486387:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018812.486395:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018812.486431:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247333044480/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018812.486466:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018812.486471:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018812.486473:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018812.486477:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3075 00000100:00080000:2.0:1734018812.486481:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018812.486485:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018812.486495:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018812.486505:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247333044608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018812.486515:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018812.486518:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018812.486520:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018812.486525:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3075 00000100:00080000:2.0:1734018812.486527:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018812.486531:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018812.486537:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018812.486545:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247333044736/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018812.486554:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018812.486557:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018812.486558:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018812.486561:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3075 00000100:00080000:2.0:1734018812.486564:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018812.486567:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018812.486571:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018812.486578:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333044480:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734018812.486580:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333044864/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018812.486589:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018812.486592:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018812.486594:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018812.486597:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3075 00000100:00080000:2.0:1734018812.486600:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018812.486603:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018812.486607:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018812.486615:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333044992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018812.486623:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018812.486626:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018812.486629:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018812.486633:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3075 00000100:00080000:2.0:1734018812.486635:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018812.486639:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018812.486643:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018812.486646:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018812.486651:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333045120/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018812.486659:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018812.486662:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018812.486663:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018812.486667:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3075 00000100:00100000:1.0:1734018812.486668:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018812/real 1734018812] req@ffff8800a73c9c00 x1818247333044480/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018812.486669:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018812.486672:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018812.486677:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018812.486684:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333045248/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018812.486695:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247333044480/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018812.486705:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018812.486708:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018812.486713:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333044480:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018812.486721:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333044608:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018812.486738:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018812.486745:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018812/real 1734018812] req@ffff8800a73c9500 x1818247333044608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018812.486755:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247333044608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018812.486763:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018812.486765:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018812.486769:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333044608:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018812.486774:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333044736:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018812.486788:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018812.486793:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018812/real 1734018812] req@ffff8800a73c8000 x1818247333044736/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018812.486803:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247333044736/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018812.486811:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018812.486814:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018812.486817:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333044736:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018812.486842:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333044864:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018812.486855:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018812.486862:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018812/real 1734018812] req@ffff8800a73c8e00 x1818247333044864/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018812.486872:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333044864/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018812.486879:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018812.486881:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018812.486884:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333044864:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018812.486890:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333044992:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018812.486901:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018812.486907:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018812/real 1734018812] req@ffff8800a73c9880 x1818247333044992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018812.486917:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333044992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018812.486924:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018812.486926:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018812.486929:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333044992:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018812.486935:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333045120:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018812.486950:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018812.486956:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018812/real 1734018812] req@ffff8800a73cb100 x1818247333045120/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018812.486966:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333045120/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018812.486973:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018812.486975:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018812.486978:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333045120:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018812.486983:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333045248:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018812.486995:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018812.487000:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018812/real 1734018812] req@ffff8800a73cad80 x1818247333045248/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018812.487010:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333045248/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018812.487017:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018812.487019:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018812.487022:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333045248:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018813.330523:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018814.332550:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018815.334510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018816.336524:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018817.338450:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018817.494330:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018817.494339:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018817.494343:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018817.494353:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3080 00000100:00080000:2.0:1734018817.494363:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018817.494371:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018817.494408:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247333045376/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018817.494449:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018817.494453:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018817.494455:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018817.494460:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3080 00000100:00080000:2.0:1734018817.494463:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018817.494467:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018817.494477:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018817.494487:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247333045504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018817.494497:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018817.494500:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018817.494502:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018817.494506:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3080 00000100:00080000:2.0:1734018817.494509:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018817.494512:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018817.494517:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018817.494526:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247333045632/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018817.494534:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018817.494537:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018817.494538:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018817.494542:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3080 00000100:00080000:2.0:1734018817.494544:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018817.494548:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018817.494553:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018817.494558:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333045376:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734018817.494562:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333045760/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018817.494571:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018817.494574:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018817.494576:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018817.494579:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3080 00000100:00080000:2.0:1734018817.494581:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018817.494585:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018817.494589:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018817.494597:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333045888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018817.494605:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018817.494608:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018817.494610:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018817.494614:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3080 00000100:00080000:2.0:1734018817.494616:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018817.494620:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018817.494624:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018817.494625:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018817.494631:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333046016/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018817.494639:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018817.494643:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018817.494645:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018817.494647:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018817/real 1734018817] req@ffff8800a73ca680 x1818247333045376/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018817.494649:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3080 00000100:00080000:2.0:1734018817.494652:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018817.494655:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018817.494659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018817.494666:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333046144/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018817.494675:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ca680 x1818247333045376/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018817.494686:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018817.494689:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018817.494694:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333045376:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018817.494701:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333045504:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018817.494719:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018817.494725:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018817/real 1734018817] req@ffff8800a73c9f80 x1818247333045504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018817.494736:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9f80 x1818247333045504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018817.494744:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018817.494747:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018817.494750:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333045504:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018817.494756:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333045632:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018817.494770:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018817.494775:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018817/real 1734018817] req@ffff8800a73caa00 x1818247333045632/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018817.494785:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73caa00 x1818247333045632/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018817.494793:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018817.494796:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018817.494799:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333045632:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018817.494823:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333045760:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018817.494837:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018817.494843:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018817/real 1734018817] req@ffff8800a73cad80 x1818247333045760/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018817.494854:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333045760/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018817.494861:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018817.494863:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018817.494866:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333045760:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018817.494871:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333045888:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018817.494883:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018817.494889:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018817/real 1734018817] req@ffff8800a73cb100 x1818247333045888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018817.494899:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333045888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018817.494907:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018817.494909:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018817.494912:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333045888:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018817.494917:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333046016:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018817.494932:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018817.494938:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018817/real 1734018817] req@ffff8800a73c9880 x1818247333046016/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018817.494948:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333046016/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018817.494955:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018817.494957:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018817.494960:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333046016:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018817.494965:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333046144:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018817.494976:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018817.494982:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018817/real 1734018817] req@ffff8800a73c8e00 x1818247333046144/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018817.494992:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333046144/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018817.494999:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018817.495001:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018817.495004:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333046144:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018818.340515:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018819.342490:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018820.344473:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018821.346476:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018822.348473:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018822.502332:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018822.502341:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018822.502346:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018822.502356:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3085 00000100:00080000:2.0:1734018822.502366:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018822.502374:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018822.502410:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247333046272/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018822.502442:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018822.502447:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018822.502449:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018822.502454:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3085 00000100:00080000:2.0:1734018822.502457:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018822.502461:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018822.502471:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018822.502482:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247333046400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018822.502493:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018822.502496:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018822.502498:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018822.502501:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3085 00000100:00080000:2.0:1734018822.502504:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018822.502507:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018822.502512:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018822.502520:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247333046528/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018822.502528:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018822.502531:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018822.502533:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018822.502536:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3085 00000100:00080000:2.0:1734018822.502538:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018822.502542:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018822.502546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018822.502550:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333046272:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734018822.502555:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333046656/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018822.502564:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018822.502567:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018822.502569:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018822.502572:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3085 00000100:00080000:2.0:1734018822.502575:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018822.502579:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018822.502583:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018822.502591:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333046784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018822.502599:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018822.502602:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018822.502604:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018822.502608:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3085 00000100:00080000:2.0:1734018822.502610:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018822.502614:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018822.502617:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:2.0:1734018822.502618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018822.502625:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333046912/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018822.502633:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018822.502636:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018822.502638:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018822.502639:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018822/real 1734018822] req@ffff8800a73c8000 x1818247333046272/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018822.502641:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3085 00000100:00080000:2.0:1734018822.502644:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018822.502647:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018822.502651:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018822.502659:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333047040/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018822.502666:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247333046272/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018822.502676:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018822.502680:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018822.502684:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333046272:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018822.502692:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333046400:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018822.502709:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018822.502716:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018822/real 1734018822] req@ffff8800a73c9500 x1818247333046400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018822.502727:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247333046400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018822.502734:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018822.502737:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018822.502740:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333046400:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018822.502746:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333046528:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018822.502759:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018822.502765:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018822/real 1734018822] req@ffff8800a73c9c00 x1818247333046528/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018822.502775:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247333046528/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018822.502782:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018822.502785:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018822.502789:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333046528:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018822.502813:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333046656:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018822.502826:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018822.502832:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018822/real 1734018822] req@ffff8800a73c8e00 x1818247333046656/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018822.502842:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333046656/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018822.502849:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018822.502852:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018822.502855:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333046656:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018822.502859:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333046784:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018822.502871:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018822.502877:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018822/real 1734018822] req@ffff8800a73c9880 x1818247333046784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018822.502886:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333046784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018822.502894:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018822.502896:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018822.502899:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333046784:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018822.502904:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333046912:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018822.502919:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018822.502925:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018822/real 1734018822] req@ffff8800a73cb100 x1818247333046912/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018822.502935:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333046912/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018822.502942:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018822.502944:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018822.502947:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333046912:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018822.502952:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333047040:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018822.502964:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018822.502970:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018822/real 1734018822] req@ffff8800a73cad80 x1818247333047040/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018822.502979:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333047040/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018822.502986:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018822.502988:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018822.502991:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333047040:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018823.350507:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018824.352502:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018825.354503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018826.356441:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018827.358477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018827.510351:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018827.510361:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018827.510365:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018827.510375:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3090 00000100:00080000:2.0:1734018827.510385:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018827.510392:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018827.510429:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247333047168/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018827.510463:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018827.510470:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018827.510473:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018827.510479:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3090 00000100:00080000:2.0:1734018827.510484:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018827.510490:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018827.510503:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018827.510522:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247333047296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018827.510527:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333047168:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734018827.510538:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018827.510542:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018827.510545:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018827.510550:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3090 00000100:00080000:2.0:1734018827.510554:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018827.510558:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018827.510566:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018827.510580:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247333047424/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018827.510592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018827.510597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1734018827.510599:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:2.0:1734018827.510600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018827.510605:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3090 00000100:00080000:2.0:1734018827.510608:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018827.510613:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018827.510619:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018827.510623:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018827/real 1734018827] req@ffff8800a73caa00 x1818247333047168/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018827.510633:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333047552/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018827.510647:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018827.510652:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018827.510654:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018827.510658:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73caa00 x1818247333047168/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018827.510659:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3090 00000100:00080000:2.0:1734018827.510663:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018827.510669:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:1.0:1734018827.510672:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1734018827.510675:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1734018827.510676:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018827.510683:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333047168:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734018827.510687:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333047680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018827.510699:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018827.510703:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018827.510706:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018827.510710:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333047296:192.168.201.152@tcp:38:kworker.0 00000100:00080000:2.0:1734018827.510712:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3090 00000100:00080000:2.0:1734018827.510716:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018827.510721:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018827.510728:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018827.510736:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018827.510740:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333047808/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018827.510747:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018827/real 1734018827] req@ffff8800a73c9f80 x1818247333047296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018827.510755:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018827.510761:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018827.510764:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018827.510765:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9f80 x1818247333047296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018827.510769:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3090 00000100:00080000:2.0:1734018827.510773:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1734018827.510778:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734018827.510779:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:1.0:1734018827.510782:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000008:00100000:2.0:1734018827.510786:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018827.510787:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333047296:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018827.510796:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333047424:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734018827.510798:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333047936/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018827.510818:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018827.510829:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018827/real 1734018827] req@ffff8800a73ca680 x1818247333047424/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018827.510845:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ca680 x1818247333047424/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018827.510858:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018827.510863:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018827.510868:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333047424:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018827.510877:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333047552:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018827.510901:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018827.510910:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018827/real 1734018827] req@ffff8800a73cad80 x1818247333047552/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018827.510956:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333047552/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018827.510964:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018827.510967:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018827.510970:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333047552:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018827.510976:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333047680:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018827.510992:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018827.510999:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018827/real 1734018827] req@ffff8800a73cb100 x1818247333047680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018827.511009:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333047680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018827.511017:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018827.511019:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018827.511022:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333047680:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018827.511039:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333047808:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018827.511055:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018827.511062:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018827/real 1734018827] req@ffff8800a73c9880 x1818247333047808/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018827.511072:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333047808/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018827.511080:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018827.511082:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018827.511085:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333047808:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018827.511090:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333047936:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018827.511101:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018827.511107:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018827/real 1734018827] req@ffff8800a73c8e00 x1818247333047936/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018827.511116:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333047936/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018827.511123:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018827.511125:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018827.511128:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333047936:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018828.360475:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018829.362513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018830.364426:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018831.366503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018832.368464:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018832.518348:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018832.518358:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018832.518363:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018832.518373:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3095 00000100:00080000:2.0:1734018832.518382:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018832.518390:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018832.518427:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247333048064/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018832.518470:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018832.518476:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018832.518479:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018832.518486:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3095 00000100:00080000:2.0:1734018832.518491:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018832.518497:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018832.518510:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018832.518530:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247333048192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018832.518538:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333048064:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734018832.518546:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018832.518550:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018832.518552:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018832.518557:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3095 00000100:00080000:2.0:1734018832.518561:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018832.518566:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018832.518573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018832.518586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247333048320/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018832.518598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018832.518602:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018832.518604:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734018832.518609:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:2.0:1734018832.518609:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3095 00000100:00080000:2.0:1734018832.518613:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018832.518619:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018832.518625:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018832.518633:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018832/real 1734018832] req@ffff8800a73c9c00 x1818247333048064/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018832.518639:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333048448/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018832.518653:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018832.518658:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018832.518660:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018832.518665:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3095 00000100:00100000:1.0:1734018832.518668:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247333048064/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018832.518669:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018832.518675:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018832.518682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1734018832.518683:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018832.518687:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018832.518694:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333048064:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734018832.518694:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333048576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018832.518708:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018832.518712:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018832.518714:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018832.518720:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333048192:192.168.201.152@tcp:38:kworker.0 00000100:00080000:2.0:1734018832.518720:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3095 00000100:00080000:2.0:1734018832.518724:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018832.518729:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018832.518735:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018832.518746:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734018832.518747:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333048704/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018832.518758:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018832/real 1734018832] req@ffff8800a73c9500 x1818247333048192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018832.518762:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018832.518768:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018832.518771:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018832.518775:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247333048192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018832.518776:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3095 00000100:00080000:2.0:1734018832.518780:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018832.518785:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:1.0:1734018832.518788:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018832.518792:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000008:00100000:2.0:1734018832.518792:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018832.518797:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333048192:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734018832.518804:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333048832/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018832.518806:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333048320:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018832.518890:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018832.518905:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018832/real 1734018832] req@ffff8800a73c8000 x1818247333048320/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018832.518924:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247333048320/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018832.518937:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018832.518941:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018832.518947:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333048320:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018832.518954:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333048448:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018832.518987:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018832.518995:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018832/real 1734018832] req@ffff8800a73c8e00 x1818247333048448/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018832.519006:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333048448/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018832.519014:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018832.519016:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018832.519019:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333048448:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018832.519024:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333048576:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018832.519036:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018832.519042:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018832/real 1734018832] req@ffff8800a73c9880 x1818247333048576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018832.519052:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333048576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018832.519061:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018832.519063:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018832.519066:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333048576:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018832.519083:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333048704:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018832.519099:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018832.519105:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018832/real 1734018832] req@ffff8800a73cb100 x1818247333048704/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018832.519116:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333048704/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018832.519123:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018832.519125:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018832.519128:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333048704:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018832.519133:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333048832:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018832.519144:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018832.519150:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018832/real 1734018832] req@ffff8800a73cad80 x1818247333048832/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018832.519160:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333048832/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018832.519167:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018832.519169:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018832.519172:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333048832:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018833.370533:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018834.372516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018835.374494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018836.376479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018837.378481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018837.526498:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018837.526509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018837.526515:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018837.526528:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3100 00000100:00080000:0.0:1734018837.526540:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018837.526549:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018837.526598:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333048960/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018837.526642:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018837.526647:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018837.526649:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018837.526654:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3100 00000100:00080000:0.0:1734018837.526657:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018837.526662:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018837.526675:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018837.526688:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333049088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018837.526698:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018837.526703:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018837.526704:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018837.526708:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3100 00000100:00080000:0.0:1734018837.526711:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018837.526715:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018837.526720:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018837.526731:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333049216/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018837.526740:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018837.526773:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018837.526775:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018837.526780:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3100 00000100:00080000:0.0:1734018837.526783:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018837.526787:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018837.526792:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018837.526795:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333048960:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018837.526802:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333049344/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018837.526813:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018837.526816:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018837.526818:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018837.526823:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3100 00000100:00080000:0.0:1734018837.526826:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018837.526829:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018837.526834:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018837.526842:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333049472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018837.526851:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018837.526854:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018837.526856:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018837.526860:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3100 00000100:00080000:0.0:1734018837.526863:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018837.526867:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018837.526871:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018837.526880:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333049600/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018837.526888:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018837.526891:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018837.526893:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018837.526895:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333049088:192.168.201.152@tcp:38:kworker.0 00000100:00080000:0.0:1734018837.526896:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3100 00000100:00080000:0.0:1734018837.526899:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018837.526903:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018837.526908:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018837.526917:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333049728/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018837.526919:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333049216:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018837.526945:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333049344:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018837.526961:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333049472:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018837.526978:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333049600:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018837.526994:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333049728:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018838.380490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018839.382510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018840.384510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1734018840.532542:0:1957:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:2.0:1734018840.532597:0:1957:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1734018840.532618:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018840.532653:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018840.532662:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018840.532669:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018840.532675:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018840.532681:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:2.0:1734018840.532687:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734018840.532789:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018837/real 1734018840] req@ffff8800a705f800 x1818247333049344/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018840.532819:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333049344/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018840.532830:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018840.532835:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018840.532840:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333049344:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018840.532848:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018837/real 1734018840] req@ffff8800a705d500 x1818247333049472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018840.532860:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333049472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018840.532868:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018840.532870:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018840.532874:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333049472:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018840.532880:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018837/real 1734018840] req@ffff8800a705e680 x1818247333049600/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018840.532891:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333049600/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018840.532899:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018840.532902:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018840.532905:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333049600:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018840.532911:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018837/real 1734018840] req@ffff8800a705c700 x1818247333049728/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018840.532922:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333049728/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018840.532929:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018840.532931:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018840.532934:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333049728:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018840.532940:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018837/real 1734018840] req@ffff8800a705df80 x1818247333048960/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018840.532951:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333048960/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018840.532958:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018840.532961:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018840.532964:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333048960:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018840.532969:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018837/real 1734018840] req@ffff8800a705ca80 x1818247333049088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018840.532979:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333049088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018840.532994:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018840.532997:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018840.533000:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333049088:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018840.533005:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018837/real 1734018840] req@ffff8800a705d880 x1818247333049216/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018840.533016:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333049216/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018840.533023:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018840.533026:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018840.533029:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333049216:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018841.386510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018842.388506:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018842.534365:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018842.534376:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018842.534383:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018842.534397:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3105 00000100:00080000:0.0:1734018842.534409:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018842.534420:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018842.534468:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333049856/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018842.534507:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018842.534512:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018842.534514:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018842.534519:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3105 00000100:00080000:0.0:1734018842.534523:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018842.534527:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018842.534539:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018842.534551:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333049984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018842.534561:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018842.534564:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018842.534566:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018842.534571:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3105 00000100:00080000:0.0:1734018842.534574:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018842.534577:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018842.534583:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018842.534593:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333050112/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018842.534603:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018842.534606:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018842.534608:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018842.534612:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3105 00000100:00080000:0.0:1734018842.534614:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018842.534618:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018842.534622:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018842.534631:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333050240/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018842.534636:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333049856:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018842.534641:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018842.534645:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018842.534646:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018842.534650:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3105 00000100:00080000:0.0:1734018842.534652:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018842.534656:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018842.534694:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018842.534705:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333050368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018842.534715:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018842.534719:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018842.534721:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018842.534726:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3105 00000100:00080000:0.0:1734018842.534738:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018842.534742:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018842.534747:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734018842.534748:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018842.534758:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333050496/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018842.534767:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018842.534771:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734018842.534772:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018842/real 1734018842] req@ffff8800a705c380 x1818247333049856/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018842.534773:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018842.534777:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3105 00000100:00080000:0.0:1734018842.534780:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018842.534784:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018842.534788:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018842.534793:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333049856/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018842.534797:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333050624/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018842.534805:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018842.534809:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018842.534814:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333049856:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018842.534823:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333049984:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018842.534843:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018842.534850:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018842/real 1734018842] req@ffff8800a705d180 x1818247333049984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018842.534862:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333049984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018842.534870:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018842.534872:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018842.534876:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333049984:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018842.534883:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333050112:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018842.534897:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018842.534903:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018842/real 1734018842] req@ffff8800a705ed80 x1818247333050112/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018842.534913:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333050112/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018842.534920:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018842.534922:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018842.534925:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333050112:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018842.534952:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333050240:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018842.534965:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018842.534971:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018842/real 1734018842] req@ffff8800a705e300 x1818247333050240/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018842.534981:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333050240/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018842.534997:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018842.534999:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018842.535003:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333050240:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018842.535008:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333050368:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018842.535021:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018842.535027:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018842/real 1734018842] req@ffff8800a705c000 x1818247333050368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018842.535037:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333050368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018842.535044:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018842.535047:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018842.535050:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333050368:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018842.535056:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333050496:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018842.535069:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018842.535074:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018842/real 1734018842] req@ffff8800a705dc00 x1818247333050496/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018842.535084:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333050496/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018842.535091:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018842.535094:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018842.535097:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333050496:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018842.535103:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333050624:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018842.535116:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018842.535122:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018842/real 1734018842] req@ffff8800a705f100 x1818247333050624/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018842.535132:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333050624/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018842.535139:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018842.535141:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018842.535144:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333050624:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018843.390354:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018844.392452:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018845.394487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018846.396449:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018847.398498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018847.542349:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018847.542360:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018847.542365:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018847.542380:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3110 00000100:00080000:0.0:1734018847.542391:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018847.542401:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018847.542449:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333050752/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018847.542497:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018847.542501:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018847.542503:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018847.542509:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3110 00000100:00080000:0.0:1734018847.542512:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018847.542516:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018847.542528:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018847.542540:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333050880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018847.542550:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018847.542553:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018847.542555:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018847.542559:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3110 00000100:00080000:0.0:1734018847.542562:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018847.542566:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018847.542572:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018847.542620:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333051008/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018847.542629:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018847.542632:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018847.542634:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018847.542638:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3110 00000100:00080000:0.0:1734018847.542641:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018847.542646:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018847.542651:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018847.542652:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333050752:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018847.542660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333051136/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018847.542671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018847.542674:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018847.542676:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018847.542680:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3110 00000100:00080000:0.0:1734018847.542683:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018847.542687:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018847.542691:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018847.542700:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333051264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018847.542708:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018847.542712:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018847.542713:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018847.542717:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3110 00000100:00080000:0.0:1734018847.542720:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018847.542724:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018847.542729:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018847.542734:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018847.542737:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333051392/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018847.542745:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018847.542750:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018847.542751:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018847.542755:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3110 00000100:00080000:0.0:1734018847.542758:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018847.542762:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018847.542764:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018847/real 1734018847] req@ffff8800a705fb80 x1818247333050752/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018847.542766:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018847.542775:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333051520/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018847.542793:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333050752/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018847.542811:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018847.542816:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018847.542823:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333050752:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018847.542832:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333050880:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018847.542861:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018847.542870:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018847/real 1734018847] req@ffff8800a705ea00 x1818247333050880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018847.542890:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333050880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018847.542898:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018847.542901:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018847.542904:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333050880:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018847.542911:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333051008:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018847.542925:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018847.542931:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018847/real 1734018847] req@ffff8800a705f100 x1818247333051008/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018847.542941:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333051008/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018847.542948:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018847.542951:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018847.542954:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333051008:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018847.542980:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333051136:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018847.542993:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018847.542999:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018847/real 1734018847] req@ffff8800a705dc00 x1818247333051136/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018847.543010:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333051136/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018847.543017:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018847.543020:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018847.543023:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333051136:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018847.543030:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333051264:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018847.543043:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018847.543049:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018847/real 1734018847] req@ffff8800a705c000 x1818247333051264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018847.543059:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333051264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018847.543067:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018847.543069:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018847.543072:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333051264:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018847.543078:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333051392:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018847.543090:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018847.543096:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018847/real 1734018847] req@ffff8800a705e300 x1818247333051392/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018847.543106:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333051392/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018847.543113:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018847.543115:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018847.543118:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333051392:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018847.543124:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333051520:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018847.543139:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018847.543145:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018847/real 1734018847] req@ffff8800a705ed80 x1818247333051520/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018847.543155:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333051520/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018847.543163:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018847.543165:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018847.543168:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333051520:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018848.400522:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018849.402494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018850.404476:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018851.406477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018852.408528:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018852.550349:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018852.550359:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018852.550365:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018852.550378:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3115 00000100:00080000:0.0:1734018852.550389:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018852.550399:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018852.550445:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333051648/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018852.550484:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018852.550521:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018852.550524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018852.550530:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3115 00000100:00080000:0.0:1734018852.550534:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018852.550539:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018852.550551:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018852.550564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333051776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018852.550576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018852.550580:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018852.550581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018852.550585:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3115 00000100:00080000:0.0:1734018852.550588:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018852.550591:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018852.550596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018852.550606:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333051904/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018852.550611:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333051648:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018852.550616:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018852.550620:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018852.550621:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018852.550625:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3115 00000100:00080000:0.0:1734018852.550628:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018852.550631:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018852.550636:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018852.550645:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333052032/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018852.550653:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018852.550657:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018852.550658:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018852.550662:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3115 00000100:00080000:0.0:1734018852.550675:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018852.550679:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018852.550684:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1734018852.550687:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018852.550694:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333052160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018852.550702:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018852.550706:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018852.550707:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018852.550711:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018852/real 1734018852] req@ffff8800a705d180 x1818247333051648/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018852.550712:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3115 00000100:00080000:0.0:1734018852.550715:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018852.550719:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018852.550723:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018852.550732:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333052288/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018852.550732:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333051648/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018852.550740:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018852.550744:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1734018852.550744:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734018852.550745:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734018852.550747:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:0.0:1734018852.550749:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3115 00000100:00080000:0.0:1734018852.550751:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018852.550752:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333051648:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018852.550755:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018852.550760:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018852.550761:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333051776:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018852.550771:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333052416/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018852.550779:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018852.550786:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018852/real 1734018852] req@ffff8800a705c380 x1818247333051776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018852.550798:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333051776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018852.550807:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018852.550810:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018852.550814:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333051776:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018852.550838:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333051904:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018852.550853:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018852.550859:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018852/real 1734018852] req@ffff8800a705d880 x1818247333051904/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018852.550870:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333051904/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018852.550878:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018852.550880:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018852.550884:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333051904:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018852.550890:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333052032:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018852.550902:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018852.550907:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018852/real 1734018852] req@ffff8800a705ca80 x1818247333052032/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018852.550917:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333052032/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018852.550925:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018852.550927:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018852.550930:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333052032:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018852.550936:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333052160:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018852.550949:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018852.550955:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018852/real 1734018852] req@ffff8800a705df80 x1818247333052160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018852.550966:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333052160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018852.550973:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018852.550976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018852.550979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333052160:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018852.550985:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333052288:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018852.550997:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018852.551003:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018852/real 1734018852] req@ffff8800a705c700 x1818247333052288/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018852.551013:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333052288/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018852.551020:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018852.551022:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018852.551025:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333052288:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018852.551031:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333052416:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018852.551043:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018852.551049:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018852/real 1734018852] req@ffff8800a705e680 x1818247333052416/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018852.551060:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333052416/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018852.551068:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018852.551070:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018852.551073:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333052416:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018853.410531:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018854.412450:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018855.414526:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018856.416510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018857.418504:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018857.558354:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018857.558366:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018857.558373:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018857.558387:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3120 00000100:00080000:0.0:1734018857.558398:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018857.558442:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018857.558491:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333052544/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018857.558528:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018857.558533:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018857.558535:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018857.558540:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3120 00000100:00080000:0.0:1734018857.558543:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018857.558548:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018857.558561:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018857.558573:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333052672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018857.558583:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018857.558587:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018857.558589:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018857.558593:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3120 00000100:00080000:0.0:1734018857.558596:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018857.558600:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018857.558606:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018857.558618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333052800/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018857.558628:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018857.558631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018857.558632:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018857.558636:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3120 00000100:00080000:0.0:1734018857.558638:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018857.558642:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018857.558646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018857.558655:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333052928/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018857.558663:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333052544:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018857.558665:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018857.558669:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018857.558671:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018857.558674:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3120 00000100:00080000:0.0:1734018857.558677:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018857.558680:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018857.558685:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018857.558694:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333053056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018857.558702:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018857.558705:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018857.558708:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018857.558712:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3120 00000100:00080000:0.0:1734018857.558725:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018857.558729:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018857.558734:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018857.558738:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018857.558743:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333053184/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018857.558751:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018857.558755:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018857.558756:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018857.558760:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3120 00000100:00080000:0.0:1734018857.558763:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018857.558763:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018857/real 1734018857] req@ffff8800a705d500 x1818247333052544/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018857.558767:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018857.558771:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018857.558780:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333053312/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018857.558783:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333052544/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018857.558796:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018857.558800:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018857.558805:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333052544:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018857.558813:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333052672:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018857.558832:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018857.558838:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018857/real 1734018857] req@ffff8800a705f800 x1818247333052672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018857.558850:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333052672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018857.558858:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018857.558861:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018857.558865:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333052672:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018857.558872:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333052800:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018857.558886:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018857.558892:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018857/real 1734018857] req@ffff8800a705e680 x1818247333052800/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018857.558902:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333052800/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018857.558910:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018857.558912:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018857.558916:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333052800:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018857.558942:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333052928:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018857.558956:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018857.558962:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018857/real 1734018857] req@ffff8800a705c700 x1818247333052928/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018857.558972:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333052928/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018857.558980:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018857.558983:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018857.558986:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333052928:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018857.558992:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333053056:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018857.559008:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018857.559014:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018857/real 1734018857] req@ffff8800a705df80 x1818247333053056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018857.559025:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333053056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018857.559032:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018857.559035:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018857.559038:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333053056:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018857.559044:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333053184:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018857.559056:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018857.559062:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018857/real 1734018857] req@ffff8800a705ca80 x1818247333053184/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018857.559072:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333053184/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018857.559079:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018857.559081:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018857.559084:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333053184:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018857.559089:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333053312:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018857.559102:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018857.559108:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018857/real 1734018857] req@ffff8800a705d880 x1818247333053312/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018857.559124:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333053312/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018857.559132:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018857.559134:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018857.559137:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333053312:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018858.420558:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018859.422523:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018860.424545:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018861.426521:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018862.428484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018862.566393:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018862.566402:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018862.566408:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018862.566422:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3125 00000100:00080000:0.0:1734018862.566433:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018862.566443:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018862.566491:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333053440/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018862.566536:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018862.566541:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018862.566543:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018862.566548:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3125 00000100:00080000:0.0:1734018862.566551:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018862.566556:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018862.566567:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018862.566580:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333053568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018862.566589:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018862.566593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018862.566594:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018862.566598:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3125 00000100:00080000:0.0:1734018862.566601:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018862.566605:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018862.566610:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018862.566619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333053696/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018862.566627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018862.566630:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018862.566632:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018862.566635:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3125 00000100:00080000:0.0:1734018862.566638:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018862.566642:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018862.566647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018862.566656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333053824/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018862.566661:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333053440:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018862.566666:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018862.566670:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018862.566671:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018862.566675:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3125 00000100:00080000:0.0:1734018862.566678:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018862.566681:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018862.566686:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018862.566694:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333053952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018862.566702:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018862.566706:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018862.566707:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018862.566711:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3125 00000100:00080000:0.0:1734018862.566724:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018862.566728:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018862.566733:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018862.566738:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018862.566742:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333054080/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018862.566750:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018862.566754:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018862.566756:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018862.566760:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3125 00000100:00080000:0.0:1734018862.566763:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018862.566763:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018862/real 1734018862] req@ffff8800a705c380 x1818247333053440/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018862.566767:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018862.566772:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018862.566782:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333054208/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018862.566783:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333053440/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018862.566796:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018862.566800:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018862.566804:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333053440:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018862.566813:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333053568:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018862.566832:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018862.566839:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018862/real 1734018862] req@ffff8800a705d180 x1818247333053568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018862.566850:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333053568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018862.566858:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018862.566862:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018862.566865:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333053568:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018862.566872:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333053696:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018862.566887:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018862.566893:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018862/real 1734018862] req@ffff8800a705ed80 x1818247333053696/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018862.566903:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333053696/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018862.566911:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018862.566913:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018862.566917:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333053696:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018862.566944:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333053824:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018862.566956:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018862.566962:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018862/real 1734018862] req@ffff8800a705e300 x1818247333053824/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018862.566973:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333053824/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018862.566981:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018862.566983:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018862.566986:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333053824:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018862.566992:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333053952:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018862.567009:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018862.567015:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018862/real 1734018862] req@ffff8800a705c000 x1818247333053952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018862.567026:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333053952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018862.567033:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018862.567035:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018862.567038:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333053952:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018862.567044:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333054080:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018862.567056:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018862.567062:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018862/real 1734018862] req@ffff8800a705dc00 x1818247333054080/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018862.567073:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333054080/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018862.567080:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018862.567083:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018862.567086:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333054080:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018862.567092:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333054208:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018862.567104:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018862.567110:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018862/real 1734018862] req@ffff8800a705f100 x1818247333054208/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018862.567126:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333054208/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018862.567134:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018862.567136:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018862.567139:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333054208:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018863.430500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018864.432507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018865.434503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018866.436492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018867.438503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018867.574444:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018867.574453:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018867.574458:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018867.574468:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3130 00000100:00080000:2.0:1734018867.574477:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018867.574484:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018867.574520:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247333054336/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018867.574566:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018867.574571:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018867.574572:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018867.574577:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3130 00000100:00080000:2.0:1734018867.574580:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018867.574584:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018867.574593:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018867.574604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247333054464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018867.574614:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018867.574618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018867.574620:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018867.574623:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3130 00000100:00080000:2.0:1734018867.574626:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018867.574629:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018867.574634:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018867.574643:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247333054592/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018867.574652:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018867.574655:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018867.574656:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018867.574659:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3130 00000100:00080000:2.0:1734018867.574662:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018867.574666:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018867.574671:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333054336:192.168.201.152@tcp:250:kworker.0 00000008:00100000:2.0:1734018867.574671:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018867.574679:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333054720/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018867.574688:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018867.574691:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018867.574693:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018867.574696:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3130 00000100:00080000:2.0:1734018867.574698:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018867.574701:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018867.574706:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018867.574713:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333054848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018867.574721:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018867.574724:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018867.574726:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018867.574729:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3130 00000100:00080000:2.0:1734018867.574732:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018867.574736:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018867.574739:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:2.0:1734018867.574740:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018867.574748:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333054976/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018867.574757:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018867.574760:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734018867.574761:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018867/real 1734018867] req@ffff8800a73ca680 x1818247333054336/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018867.574762:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018867.574766:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3130 00000100:00080000:2.0:1734018867.574769:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018867.574772:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018867.574776:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018867.574780:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ca680 x1818247333054336/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018867.574784:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333055104/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018867.574791:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018867.574794:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018867.574799:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333054336:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018867.574807:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333054464:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018867.574824:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018867.574830:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018867/real 1734018867] req@ffff8800a73c9f80 x1818247333054464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018867.574841:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9f80 x1818247333054464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018867.574849:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018867.574852:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018867.574856:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333054464:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018867.574862:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333054592:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018867.574876:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018867.574881:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018867/real 1734018867] req@ffff8800a73caa00 x1818247333054592/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018867.574891:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73caa00 x1818247333054592/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018867.574899:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018867.574902:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018867.574905:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333054592:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018867.574929:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333054720:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018867.574942:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018867.574947:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018867/real 1734018867] req@ffff8800a73cad80 x1818247333054720/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018867.574958:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333054720/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018867.574966:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018867.574968:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018867.574971:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333054720:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018867.574976:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333054848:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018867.574991:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018867.574997:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018867/real 1734018867] req@ffff8800a73cb100 x1818247333054848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018867.575008:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333054848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018867.575015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018867.575017:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018867.575020:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333054848:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018867.575026:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333054976:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018867.575037:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018867.575043:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018867/real 1734018867] req@ffff8800a73c9880 x1818247333054976/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018867.575053:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333054976/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018867.575060:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018867.575062:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018867.575065:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333054976:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018867.575070:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333055104:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018867.575082:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018867.575088:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018867/real 1734018867] req@ffff8800a73c8e00 x1818247333055104/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018867.575105:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333055104/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018867.575112:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018867.575114:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018867.575117:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333055104:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018868.440494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018869.442502:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018870.444473:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018871.446501:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018872.448449:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734018872.582353:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018872.582362:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734018872.582366:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018872.582376:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3135 00000100:00080000:2.0:1734018872.582386:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018872.582394:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734018872.582430:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247333055232/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018872.582463:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018872.582467:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018872.582469:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018872.582473:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3135 00000100:00080000:2.0:1734018872.582476:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018872.582480:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018872.582490:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018872.582501:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247333055360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018872.582512:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018872.582515:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018872.582517:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018872.582520:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3135 00000100:00080000:2.0:1734018872.582523:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018872.582526:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018872.582531:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734018872.582539:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247333055488/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018872.582547:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018872.582550:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018872.582552:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018872.582555:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3135 00000100:00080000:2.0:1734018872.582557:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018872.582561:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018872.582565:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018872.582568:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333055232:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734018872.582575:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333055616/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018872.582584:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018872.582587:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734018872.582588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018872.582592:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3135 00000100:00080000:2.0:1734018872.582595:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018872.582598:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018872.582603:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734018872.582610:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333055744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018872.582618:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018872.582621:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734018872.582623:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734018872.582627:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3135 00000100:00080000:2.0:1734018872.582630:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018872.582633:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018872.582636:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:2.0:1734018872.582637:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734018872.582645:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333055872/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734018872.582652:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734018872.582655:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734018872.582657:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018872.582658:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018872/real 1734018872] req@ffff8800a73c8000 x1818247333055232/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734018872.582661:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3135 00000100:00080000:2.0:1734018872.582663:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734018872.582667:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734018872.582671:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018872.582677:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247333055232/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734018872.582678:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333056000/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018872.582688:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018872.582692:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018872.582697:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333055232:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018872.582704:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333055360:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018872.582721:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018872.582728:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018872/real 1734018872] req@ffff8800a73c9500 x1818247333055360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018872.582739:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247333055360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018872.582747:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018872.582750:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018872.582753:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333055360:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018872.582760:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333055488:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018872.582773:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018872.582779:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018872/real 1734018872] req@ffff8800a73c9c00 x1818247333055488/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018872.582789:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247333055488/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018872.582797:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018872.582800:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018872.582803:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333055488:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018872.582827:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333055616:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018872.582839:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018872.582845:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018872/real 1734018872] req@ffff8800a73c8e00 x1818247333055616/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018872.582856:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333055616/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018872.582864:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018872.582866:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018872.582869:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333055616:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018872.582874:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333055744:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018872.582889:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018872.582895:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018872/real 1734018872] req@ffff8800a73c9880 x1818247333055744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018872.582906:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333055744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018872.582913:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018872.582915:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018872.582918:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333055744:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018872.582923:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333055872:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018872.582935:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018872.582940:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018872/real 1734018872] req@ffff8800a73cb100 x1818247333055872/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018872.582950:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333055872/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018872.582957:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018872.582959:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018872.582962:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333055872:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018872.582967:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333056000:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018872.582978:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018872.582984:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018872/real 1734018872] req@ffff8800a73cad80 x1818247333056000/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018872.583001:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333056000/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018872.583008:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018872.583010:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018872.583013:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333056000:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018873.450472:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018874.452440:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018875.454475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018876.456455:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018877.458492:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018877.590498:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018877.590507:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018877.590512:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018877.590523:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3140 00000100:00080000:0.0:1734018877.590533:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018877.590541:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018877.590578:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333056128/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018877.590617:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018877.590621:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018877.590623:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018877.590628:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3140 00000100:00080000:0.0:1734018877.590631:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018877.590635:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018877.590645:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018877.590656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333056256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018877.590667:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018877.590670:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018877.590672:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018877.590676:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3140 00000100:00080000:0.0:1734018877.590678:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018877.590681:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018877.590686:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018877.590696:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333056384/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018877.590705:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018877.590708:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018877.590709:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018877.590712:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3140 00000100:00080000:0.0:1734018877.590715:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018877.590718:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018877.590722:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018877.590727:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333056128:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018877.590731:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333056512/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018877.590740:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018877.590743:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018877.590745:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018877.590748:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3140 00000100:00080000:0.0:1734018877.590752:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018877.590755:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018877.590759:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018877.590767:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333056640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018877.590775:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018877.590778:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018877.590779:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018877.590783:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3140 00000100:00080000:0.0:1734018877.590786:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018877.590789:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018877.590793:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018877.590795:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018877.590801:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333056768/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018877.590809:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018877.590812:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018877.590813:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018877.590817:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3140 00000100:00100000:1.0:1734018877.590818:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018877/real 1734018877] req@ffff8800a705ea00 x1818247333056128/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018877.590819:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018877.590823:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018877.590827:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018877.590834:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333056896/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018877.590838:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333056128/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018877.590849:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018877.590853:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018877.590857:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333056128:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018877.590865:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333056256:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018877.590882:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018877.590888:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018877/real 1734018877] req@ffff8800a705fb80 x1818247333056256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018877.590899:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333056256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018877.590907:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018877.590911:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018877.590914:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333056256:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018877.590921:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333056384:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018877.590934:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018877.590940:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018877/real 1734018877] req@ffff8800a705f100 x1818247333056384/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018877.590950:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333056384/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018877.590958:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018877.590961:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018877.590964:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333056384:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018877.590988:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333056512:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018877.591001:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018877.591006:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018877/real 1734018877] req@ffff8800a705dc00 x1818247333056512/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018877.591017:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333056512/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018877.591025:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018877.591027:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018877.591030:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333056512:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018877.591035:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333056640:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018877.591051:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018877.591057:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018877/real 1734018877] req@ffff8800a705c000 x1818247333056640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018877.591110:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333056640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018877.591118:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018877.591121:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018877.591124:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333056640:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018877.591130:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333056768:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018877.591145:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018877.591151:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018877/real 1734018877] req@ffff8800a705e300 x1818247333056768/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018877.591161:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333056768/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018877.591169:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018877.591171:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018877.591174:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333056768:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018877.591179:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333056896:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018877.591191:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018877.591197:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018877/real 1734018877] req@ffff8800a705ed80 x1818247333056896/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018877.591214:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333056896/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018877.591263:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018877.591265:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018877.591268:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333056896:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018878.460518:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018879.462447:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018880.464435:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018881.466451:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018882.468493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018882.598326:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018882.598337:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018882.598342:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018882.598356:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3145 00000100:00080000:0.0:1734018882.598366:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018882.598376:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018882.598424:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333057024/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018882.598463:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018882.598467:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018882.598470:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018882.598475:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3145 00000100:00080000:0.0:1734018882.598478:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018882.598483:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018882.598494:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018882.598506:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333057152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018882.598516:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018882.598520:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018882.598522:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018882.598526:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3145 00000100:00080000:0.0:1734018882.598529:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018882.598533:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018882.598539:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018882.598548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333057280/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018882.598558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018882.598561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018882.598562:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018882.598566:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3145 00000100:00080000:0.0:1734018882.598568:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018882.598572:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018882.598576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018882.598585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333057408/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018882.598585:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333057024:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018882.598594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018882.598598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018882.598599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018882.598603:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3145 00000100:00080000:0.0:1734018882.598606:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018882.598609:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018882.598614:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018882.598622:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333057536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018882.598630:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018882.598633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018882.598636:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018882.598640:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3145 00000100:00080000:0.0:1734018882.598653:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018882.598657:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018882.598662:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018882.598664:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018882.598671:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333057664/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018882.598680:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018882.598683:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018882.598685:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018882.598688:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018882/real 1734018882] req@ffff8800a705d180 x1818247333057024/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018882.598689:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3145 00000100:00080000:0.0:1734018882.598692:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018882.598695:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018882.598700:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018882.598708:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333057024/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018882.598711:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333057792/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018882.598720:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018882.598724:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018882.598729:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333057024:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018882.598737:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333057152:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018882.598756:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018882.598763:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018882/real 1734018882] req@ffff8800a705c380 x1818247333057152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018882.598774:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333057152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018882.598782:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018882.598786:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018882.598789:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333057152:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018882.598796:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333057280:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018882.598810:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018882.598816:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018882/real 1734018882] req@ffff8800a705d880 x1818247333057280/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018882.598826:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333057280/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018882.598834:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018882.598837:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018882.598840:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333057280:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018882.598866:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333057408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018882.598879:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018882.598885:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018882/real 1734018882] req@ffff8800a705ca80 x1818247333057408/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018882.598896:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333057408/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018882.598904:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018882.598906:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018882.598909:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333057408:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018882.598915:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333057536:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018882.598931:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018882.598937:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018882/real 1734018882] req@ffff8800a705df80 x1818247333057536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018882.598947:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333057536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018882.598955:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018882.598957:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018882.598960:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333057536:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018882.598966:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333057664:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018882.599026:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018882.599035:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018882/real 1734018882] req@ffff8800a705c700 x1818247333057664/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018882.599046:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333057664/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018882.599054:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018882.599056:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018882.599059:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333057664:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018882.599065:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333057792:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018882.599082:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018882.599088:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018882/real 1734018882] req@ffff8800a705e680 x1818247333057792/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018882.599105:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333057792/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018882.599113:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018882.599115:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018882.599118:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333057792:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018883.470489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018884.472489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018885.474493:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018886.476489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018887.478492:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018887.606324:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018887.606332:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018887.606337:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018887.606347:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3150 00000100:00080000:0.0:1734018887.606357:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018887.606365:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018887.606401:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333057920/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018887.606442:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018887.606446:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018887.606448:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018887.606453:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3150 00000100:00080000:0.0:1734018887.606456:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018887.606460:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018887.606470:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018887.606481:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333058048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018887.606491:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018887.606494:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018887.606496:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018887.606499:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3150 00000100:00080000:0.0:1734018887.606502:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018887.606505:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018887.606510:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018887.606520:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333058176/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018887.606528:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018887.606531:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018887.606533:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018887.606536:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3150 00000100:00080000:0.0:1734018887.606538:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018887.606543:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018887.606547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018887.606553:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333057920:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018887.606556:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333058304/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018887.606565:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018887.606568:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018887.606569:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018887.606573:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3150 00000100:00080000:0.0:1734018887.606575:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018887.606578:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018887.606583:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018887.606590:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333058432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018887.606599:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018887.606602:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018887.606603:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018887.606607:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3150 00000100:00080000:0.0:1734018887.606610:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018887.606613:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018887.606618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018887.606623:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018887.606626:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333058560/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018887.606633:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018887.606637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018887.606639:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018887.606642:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3150 00000100:00080000:0.0:1734018887.606646:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018887.606646:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018887/real 1734018887] req@ffff8800a705f800 x1818247333057920/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018887.606649:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018887.606653:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018887.606660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333058688/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018887.606666:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333057920/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018887.606677:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018887.606680:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018887.606685:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333057920:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018887.606691:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333058048:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018887.606709:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018887.606715:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018887/real 1734018887] req@ffff8800a705d500 x1818247333058048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018887.606726:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333058048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018887.606734:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018887.606738:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018887.606741:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333058048:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018887.606747:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333058176:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018887.606761:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018887.606767:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018887/real 1734018887] req@ffff8800a705e680 x1818247333058176/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018887.606777:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333058176/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018887.606785:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018887.606787:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018887.606791:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333058176:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018887.606814:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333058304:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018887.606826:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018887.606832:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018887/real 1734018887] req@ffff8800a705c700 x1818247333058304/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018887.606843:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333058304/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018887.606851:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018887.606853:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018887.606856:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333058304:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018887.606861:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333058432:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018887.606876:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018887.606882:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018887/real 1734018887] req@ffff8800a705df80 x1818247333058432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018887.606892:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333058432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018887.606948:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018887.606950:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018887.606954:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333058432:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018887.606960:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333058560:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018887.606976:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018887.606984:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018887/real 1734018887] req@ffff8800a705ca80 x1818247333058560/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018887.606994:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333058560/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018887.607002:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018887.607004:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018887.607007:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333058560:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018887.607012:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333058688:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018887.607023:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018887.607029:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018887/real 1734018887] req@ffff8800a705d880 x1818247333058688/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018887.607045:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333058688/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018887.607052:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018887.607054:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018887.607057:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333058688:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018888.480488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018889.482441:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018890.484471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018891.486469:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018892.488455:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018892.614376:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018892.614386:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018892.614392:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018892.614407:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3155 00000100:00080000:0.0:1734018892.614418:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018892.614428:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018892.614474:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333058816/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018892.614514:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018892.614518:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018892.614521:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018892.614526:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3155 00000100:00080000:0.0:1734018892.614529:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018892.614533:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018892.614546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018892.614558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333058944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018892.614569:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018892.614573:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018892.614574:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018892.614578:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3155 00000100:00080000:0.0:1734018892.614581:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018892.614584:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018892.614589:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018892.614598:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333059072/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018892.614607:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018892.614610:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018892.614612:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018892.614615:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3155 00000100:00080000:0.0:1734018892.614618:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018892.614621:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018892.614625:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018892.614636:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333059200/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018892.614639:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333058816:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018892.614645:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018892.614649:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018892.614650:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018892.614654:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3155 00000100:00080000:0.0:1734018892.614658:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018892.614661:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018892.614666:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018892.614675:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333059328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018892.614683:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018892.614686:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018892.614688:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018892.614692:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3155 00000100:00080000:0.0:1734018892.614705:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018892.614709:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018892.614713:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018892.614716:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018892.614725:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333059456/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018892.614733:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018892.614737:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018892.614738:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018892.614741:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018892/real 1734018892] req@ffff8800a705c380 x1818247333058816/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018892.614742:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3155 00000100:00080000:0.0:1734018892.614746:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018892.614750:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018892.614754:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018892.614761:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333058816/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018892.614763:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333059584/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018892.614773:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018892.614777:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018892.614782:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333058816:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018892.614791:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333058944:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018892.614842:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018892.614850:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018892/real 1734018892] req@ffff8800a705d180 x1818247333058944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018892.614862:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333058944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018892.614871:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018892.614875:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018892.614878:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333058944:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018892.614885:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333059072:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018892.614902:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018892.614908:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018892/real 1734018892] req@ffff8800a705ed80 x1818247333059072/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018892.614918:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333059072/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018892.614926:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018892.614929:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018892.614932:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333059072:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018892.614959:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333059200:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018892.614971:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018892.614978:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018892/real 1734018892] req@ffff8800a705e300 x1818247333059200/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018892.614988:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333059200/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018892.614996:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018892.614999:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018892.615002:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333059200:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018892.615008:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333059328:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018892.615024:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018892.615030:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018892/real 1734018892] req@ffff8800a705c000 x1818247333059328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018892.615041:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333059328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018892.615048:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018892.615050:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018892.615053:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333059328:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018892.615059:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333059456:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018892.615071:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018892.615077:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018892/real 1734018892] req@ffff8800a705dc00 x1818247333059456/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018892.615088:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333059456/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018892.615095:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018892.615097:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018892.615100:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333059456:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018892.615106:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333059584:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018892.615119:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018892.615125:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018892/real 1734018892] req@ffff8800a705f100 x1818247333059584/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018892.615142:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333059584/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018892.615149:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018892.615152:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018892.615155:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333059584:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018893.490468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018894.492455:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018895.494446:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018896.496384:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018897.498349:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018897.622340:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018897.622349:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018897.622354:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018897.622364:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3160 00000100:00080000:0.0:1734018897.622374:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018897.622382:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018897.622418:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333059712/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018897.622451:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018897.622455:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018897.622457:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018897.622462:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3160 00000100:00080000:0.0:1734018897.622465:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018897.622469:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018897.622479:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018897.622490:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333059840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018897.622500:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018897.622503:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018897.622505:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018897.622509:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3160 00000100:00080000:0.0:1734018897.622511:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018897.622515:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018897.622521:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018897.622531:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333059968/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018897.622540:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018897.622543:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018897.622544:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018897.622548:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3160 00000100:00080000:0.0:1734018897.622550:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018897.622554:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018897.622558:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018897.622559:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333059712:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018897.622566:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333060096/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018897.622575:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018897.622578:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018897.622580:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018897.622583:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3160 00000100:00080000:0.0:1734018897.622586:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018897.622589:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018897.622594:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018897.622601:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333060224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018897.622609:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018897.622612:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018897.622614:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018897.622618:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3160 00000100:00080000:0.0:1734018897.622621:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018897.622624:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018897.622627:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734018897.622629:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018897.622636:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333060352/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018897.622644:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018897.622647:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018897.622649:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018897.622650:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018897/real 1734018897] req@ffff8800a705fb80 x1818247333059712/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018897.622652:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3160 00000100:00080000:0.0:1734018897.622655:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018897.622658:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018897.622662:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018897.622670:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333059712/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018897.622671:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333060480/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018897.622681:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018897.622684:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018897.622689:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333059712:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018897.622696:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333059840:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018897.622713:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018897.622720:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018897/real 1734018897] req@ffff8800a705ea00 x1818247333059840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018897.622776:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333059840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018897.622785:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018897.622789:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018897.622792:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333059840:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018897.622799:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333059968:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018897.622816:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018897.622823:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018897/real 1734018897] req@ffff8800a705f100 x1818247333059968/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018897.622833:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333059968/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018897.622841:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018897.622844:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018897.622847:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333059968:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018897.622870:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333060096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018897.622883:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018897.622889:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018897/real 1734018897] req@ffff8800a705dc00 x1818247333060096/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018897.622899:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333060096/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018897.622907:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018897.622909:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018897.622912:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333060096:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018897.622917:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333060224:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018897.622932:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018897.622938:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018897/real 1734018897] req@ffff8800a705c000 x1818247333060224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018897.622949:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333060224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018897.622956:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018897.622959:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018897.622962:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333060224:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018897.622967:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333060352:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018897.622978:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018897.622983:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018897/real 1734018897] req@ffff8800a705e300 x1818247333060352/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018897.622993:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333060352/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018897.623000:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018897.623002:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018897.623005:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333060352:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018897.623010:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333060480:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018897.623022:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018897.623028:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018897/real 1734018897] req@ffff8800a705ed80 x1818247333060480/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018897.623044:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333060480/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018897.623052:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018897.623054:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018897.623057:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333060480:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018898.500343:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018899.502345:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018900.504410:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018901.506508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018902.508484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018902.630357:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018902.630366:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018902.630370:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018902.630381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3165 00000100:00080000:0.0:1734018902.630391:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018902.630398:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018902.630433:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333060608/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018902.630475:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018902.630479:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018902.630481:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018902.630486:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3165 00000100:00080000:0.0:1734018902.630489:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018902.630493:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018902.630503:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018902.630514:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333060736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018902.630523:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018902.630527:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018902.630528:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018902.630532:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3165 00000100:00080000:0.0:1734018902.630534:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018902.630537:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018902.630542:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018902.630550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333060864/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018902.630559:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018902.630562:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018902.630563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018902.630566:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3165 00000100:00080000:0.0:1734018902.630569:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018902.630573:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018902.630577:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018902.630585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333060992/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018902.630594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018902.630597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1734018902.630598:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333060608:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018902.630599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018902.630602:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3165 00000100:00080000:0.0:1734018902.630605:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018902.630608:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018902.630612:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018902.630620:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333061120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018902.630628:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018902.630631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018902.630632:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018902.630636:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3165 00000100:00080000:0.0:1734018902.630683:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018902.630688:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018902.630693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018902.630704:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333061248/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018902.630713:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018902.630717:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018902.630719:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018902.630722:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3165 00000100:00080000:0.0:1734018902.630725:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018902.630730:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018902.630736:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018902.630743:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333060736:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734018902.630750:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333061376/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018902.630768:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333060864:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018902.630791:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333060992:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018902.630805:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333061120:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018902.630819:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333061248:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018902.630832:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333061376:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018903.510352:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018904.512434:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018905.514402:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734018905.636659:0:1958:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734018905.636713:0:1958:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734018905.636732:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018905.636771:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018905.636783:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018905.636794:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018905.636805:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018905.636815:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018905.636825:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734018905.636834:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018902/real 1734018905] req@ffff8800a705ca80 x1818247333060992/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018905.636871:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333060992/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018905.636888:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018905.636893:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018905.636900:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333060992:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018905.636912:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018902/real 1734018905] req@ffff8800a705df80 x1818247333061120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018905.636930:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333061120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018905.636939:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018905.636941:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018905.636945:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333061120:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018905.636952:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018902/real 1734018905] req@ffff8800a705c700 x1818247333061248/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018905.636967:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333061248/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018905.636978:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018905.636982:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018905.636986:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333061248:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018905.636995:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018902/real 1734018905] req@ffff8800a705e680 x1818247333061376/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018905.637011:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333061376/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018905.637023:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018905.637027:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018905.637032:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333061376:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018905.637045:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018902/real 1734018905] req@ffff8800a705d180 x1818247333060608/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018905.637066:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333060608/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018905.637077:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018905.637081:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018905.637086:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333060608:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018905.637096:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018902/real 1734018905] req@ffff8800a705c380 x1818247333060736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018905.637115:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333060736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018905.637128:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018905.637132:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018905.637137:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333060736:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018905.637147:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018902/real 1734018905] req@ffff8800a705d880 x1818247333060864/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018905.637165:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333060864/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018905.637177:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018905.637181:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018905.637186:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333060864:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018906.516482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018907.518444:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018907.638340:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018907.638349:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018907.638353:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018907.638364:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3170 00000100:00080000:0.0:1734018907.638374:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018907.638382:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018907.638417:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333061504/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018907.638458:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018907.638462:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018907.638464:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018907.638468:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3170 00000100:00080000:0.0:1734018907.638472:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018907.638476:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018907.638486:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018907.638496:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333061632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018907.638506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018907.638509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018907.638511:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018907.638514:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3170 00000100:00080000:0.0:1734018907.638517:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018907.638520:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018907.638525:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018907.638535:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333061760/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018907.638543:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018907.638546:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018907.638547:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018907.638550:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3170 00000100:00080000:0.0:1734018907.638585:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018907.638590:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018907.638595:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018907.638600:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333061504:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018907.638605:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333061888/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018907.638615:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018907.638618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018907.638620:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018907.638623:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3170 00000100:00080000:0.0:1734018907.638626:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018907.638629:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018907.638633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018907.638641:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333062016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018907.638650:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018907.638652:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018907.638654:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018907.638657:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3170 00000100:00080000:0.0:1734018907.638660:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018907.638663:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018907.638668:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018907.638671:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018907.638676:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333062144/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018907.638684:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018907.638688:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018907.638689:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018907.638693:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3170 00000100:00100000:1.0:1734018907.638693:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018907/real 1734018907] req@ffff8800a705d500 x1818247333061504/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018907.638696:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018907.638699:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018907.638704:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018907.638711:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333062272/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018907.638713:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333061504/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018907.638724:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018907.638727:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018907.638732:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333061504:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018907.638739:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333061632:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018907.638756:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018907.638763:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018907/real 1734018907] req@ffff8800a705f800 x1818247333061632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018907.638774:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333061632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018907.638781:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018907.638784:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018907.638787:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333061632:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018907.638793:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333061760:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018907.638806:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018907.638812:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018907/real 1734018907] req@ffff8800a705d880 x1818247333061760/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018907.638822:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333061760/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018907.638831:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018907.638833:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018907.638836:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333061760:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018907.638861:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333061888:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018907.638873:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018907.638879:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018907/real 1734018907] req@ffff8800a705c380 x1818247333061888/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018907.638889:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333061888/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018907.638897:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018907.638899:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018907.638902:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333061888:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018907.638907:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333062016:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018907.638918:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018907.638924:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018907/real 1734018907] req@ffff8800a705d180 x1818247333062016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018907.638934:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333062016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018907.638941:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018907.638943:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018907.638946:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333062016:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018907.638952:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333062144:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018907.638964:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018907.638970:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018907/real 1734018907] req@ffff8800a705e680 x1818247333062144/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018907.638979:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333062144/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018907.638986:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018907.638989:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018907.638992:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333062144:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018907.638996:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333062272:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018907.639008:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018907.639014:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018907/real 1734018907] req@ffff8800a705c700 x1818247333062272/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018907.639023:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333062272/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018907.639030:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018907.639032:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018907.639036:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333062272:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018908.520480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018909.522455:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018910.524453:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018911.526417:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018912.528448:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018912.646301:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018912.646311:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018912.646316:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018912.646330:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3175 00000100:00080000:0.0:1734018912.646341:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018912.646351:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018912.646399:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333062400/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018912.646436:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018912.646441:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018912.646443:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018912.646448:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3175 00000100:00080000:0.0:1734018912.646452:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018912.646456:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018912.646506:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018912.646521:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333062528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018912.646535:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018912.646539:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018912.646541:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018912.646544:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3175 00000100:00080000:0.0:1734018912.646547:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018912.646551:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018912.646556:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018912.646565:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333062656/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018912.646576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018912.646579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018912.646581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018912.646582:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333062400:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018912.646585:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3175 00000100:00080000:0.0:1734018912.646588:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018912.646592:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018912.646597:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018912.646605:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333062784/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018912.646613:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018912.646617:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018912.646618:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018912.646622:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3175 00000100:00080000:0.0:1734018912.646625:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018912.646629:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018912.646633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018912.646642:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333062912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018912.646650:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018912.646653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018912.646654:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018912.646659:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3175 00000800:00000100:1.0:1734018912.646659:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018912.646662:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018912.646666:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018912.646670:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018912.646681:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333063040/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018912.646683:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018912/real 1734018912] req@ffff8800a705df80 x1818247333062400/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018912.646689:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018912.646693:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018912.646694:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018912.646698:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3175 00000100:00080000:0.0:1734018912.646701:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018912.646704:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333062400/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018912.646705:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018912.646709:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734018912.646716:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1734018912.646717:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333063168/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734018912.646720:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018912.646725:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333062400:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018912.646734:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333062528:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018912.646753:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018912.646761:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018912/real 1734018912] req@ffff8800a705ca80 x1818247333062528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018912.646772:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333062528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018912.646780:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018912.646783:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018912.646787:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333062528:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018912.646813:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333062656:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018912.646828:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018912.646834:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018912/real 1734018912] req@ffff8800a705ed80 x1818247333062656/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018912.646844:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333062656/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018912.646852:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018912.646855:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018912.646859:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333062656:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018912.646865:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333062784:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018912.646877:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018912.646883:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018912/real 1734018912] req@ffff8800a705e300 x1818247333062784/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018912.646893:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333062784/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018912.646901:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018912.646904:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018912.646907:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333062784:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018912.646912:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333062912:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018912.646925:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018912.646931:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018912/real 1734018912] req@ffff8800a705c000 x1818247333062912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018912.646940:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333062912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018912.646948:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018912.646951:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018912.646954:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333062912:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018912.646960:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333063040:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018912.646976:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018912.646981:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018912/real 1734018912] req@ffff8800a705dc00 x1818247333063040/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018912.646992:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333063040/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018912.646999:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018912.647001:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018912.647004:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333063040:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018912.647010:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333063168:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018912.647022:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018912.647027:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018912/real 1734018912] req@ffff8800a705f100 x1818247333063168/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018912.647037:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333063168/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018912.647044:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018912.647047:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018912.647050:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333063168:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018913.530522:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018914.532505:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018915.534496:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018916.536480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018917.538503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018917.654366:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018917.654377:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018917.654424:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018917.654439:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3180 00000100:00080000:0.0:1734018917.654450:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018917.654460:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018917.654509:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333063296/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018917.654547:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018917.654552:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018917.654554:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018917.654560:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3180 00000100:00080000:0.0:1734018917.654563:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018917.654567:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018917.654579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018917.654591:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333063424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018917.654601:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018917.654605:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018917.654607:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018917.654611:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3180 00000100:00080000:0.0:1734018917.654614:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018917.654618:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018917.654625:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018917.654636:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333063552/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018917.654647:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018917.654651:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018917.654652:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018917.654656:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3180 00000100:00080000:0.0:1734018917.654659:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018917.654664:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018917.654666:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333063296:192.168.201.152@tcp:250:kworker.0 00000008:00100000:0.0:1734018917.654668:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018917.654677:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333063680/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018917.654685:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018917.654689:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018917.654690:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018917.654694:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3180 00000100:00080000:0.0:1734018917.654697:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018917.654701:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018917.654705:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018917.654714:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333063808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018917.654722:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018917.654725:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018917.654728:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018917.654732:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3180 00000100:00080000:0.0:1734018917.654735:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018917.654739:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018917.654741:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734018917.654744:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018917.654752:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333063936/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018917.654761:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018917.654764:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734018917.654764:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018917/real 1734018917] req@ffff8800a705ea00 x1818247333063296/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018917.654766:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018917.654769:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3180 00000100:00080000:0.0:1734018917.654772:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018917.654776:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018917.654780:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018917.654789:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333064064/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018917.654793:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333063296/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018917.654804:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018917.654807:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018917.654812:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333063296:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018917.654821:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333063424:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018917.654840:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018917.654847:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018917/real 1734018917] req@ffff8800a705fb80 x1818247333063424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018917.654858:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333063424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018917.654866:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018917.654869:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018917.654873:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333063424:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018917.654898:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333063552:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018917.654912:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018917.654919:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018917/real 1734018917] req@ffff8800a705f100 x1818247333063552/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018917.654929:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333063552/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018917.654937:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018917.654940:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018917.654944:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333063552:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018917.654950:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333063680:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018917.654962:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018917.654968:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018917/real 1734018917] req@ffff8800a705dc00 x1818247333063680/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018917.654978:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333063680/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018917.654986:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018917.654989:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018917.654992:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333063680:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018917.654997:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333063808:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018917.655010:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018917.655016:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018917/real 1734018917] req@ffff8800a705c000 x1818247333063808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018917.655026:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333063808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018917.655034:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018917.655036:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018917.655039:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333063808:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018917.655046:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333063936:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018917.655061:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018917.655067:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018917/real 1734018917] req@ffff8800a705e300 x1818247333063936/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018917.655078:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333063936/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018917.655085:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018917.655087:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018917.655090:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333063936:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018917.655097:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333064064:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018917.655109:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018917.655115:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018917/real 1734018917] req@ffff8800a705ed80 x1818247333064064/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018917.655125:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333064064/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018917.655132:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018917.655134:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018917.655137:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333064064:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018918.540528:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018919.542565:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018920.544513:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018921.546490:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018922.548533:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018922.662412:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018922.662422:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018922.662428:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018922.662441:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3185 00000100:00080000:0.0:1734018922.662453:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018922.662462:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018922.662510:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333064192/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018922.662555:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018922.662560:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018922.662562:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018922.662567:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3185 00000100:00080000:0.0:1734018922.662571:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018922.662575:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018922.662586:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018922.662599:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333064320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018922.662609:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018922.662612:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018922.662614:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018922.662618:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3185 00000100:00080000:0.0:1734018922.662621:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018922.662625:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018922.662629:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018922.662638:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333064448/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018922.662646:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018922.662650:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018922.662651:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018922.662655:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3185 00000100:00080000:0.0:1734018922.662657:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018922.662662:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018922.662667:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018922.662667:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333064192:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018922.662675:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333064576/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018922.662684:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018922.662687:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018922.662689:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018922.662692:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3185 00000100:00080000:0.0:1734018922.662695:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018922.662699:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018922.662703:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018922.662712:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333064704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018922.662720:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018922.662723:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018922.662724:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018922.662728:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3185 00000100:00080000:0.0:1734018922.662733:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018922.662737:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018922.662741:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018922.662749:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333064832/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018922.662752:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018922.662757:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018922.662762:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018922.662763:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018922.662767:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3185 00000100:00080000:0.0:1734018922.662771:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018922.662774:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018922.662779:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018922.662780:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018922/real 1734018922] req@ffff8800a705ca80 x1818247333064192/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018922.662790:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333064960/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018922.662810:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333064192/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018922.662821:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018922.662824:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018922.662830:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333064192:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018922.662837:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333064320:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018922.662856:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018922.662863:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018922/real 1734018922] req@ffff8800a705df80 x1818247333064320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018922.662874:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333064320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018922.662882:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018922.662884:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018922.662888:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333064320:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018922.662893:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333064448:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018922.662908:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018922.662913:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018922/real 1734018922] req@ffff8800a705c700 x1818247333064448/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018922.662924:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333064448/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018922.662931:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018922.662935:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018922.662938:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333064448:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018922.662966:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333064576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018922.662980:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018922.662985:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018922/real 1734018922] req@ffff8800a705e680 x1818247333064576/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018922.662996:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333064576/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018922.663004:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018922.663006:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018922.663009:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333064576:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018922.663013:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333064704:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018922.663025:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018922.663031:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018922/real 1734018922] req@ffff8800a705d180 x1818247333064704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018922.663041:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333064704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018922.663048:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018922.663050:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018922.663053:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333064704:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018922.663059:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333064832:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018922.663076:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018922.663082:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018922/real 1734018922] req@ffff8800a705c380 x1818247333064832/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018922.663092:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333064832/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018922.663100:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018922.663102:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018922.663104:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333064832:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018922.663109:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333064960:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018922.663122:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018922.663127:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018922/real 1734018922] req@ffff8800a705d880 x1818247333064960/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018922.663138:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333064960/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018922.663146:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018922.663148:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018922.663151:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333064960:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018923.550517:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018924.552528:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018925.554487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018926.556499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018927.558504:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018927.670355:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018927.670367:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018927.670373:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018927.670386:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3190 00000100:00080000:0.0:1734018927.670397:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018927.670407:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018927.670455:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333065088/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018927.670493:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018927.670498:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018927.670500:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018927.670505:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3190 00000100:00080000:0.0:1734018927.670508:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018927.670513:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018927.670525:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018927.670537:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333065216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018927.670549:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018927.670552:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018927.670554:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018927.670558:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3190 00000100:00080000:0.0:1734018927.670561:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018927.670564:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018927.670569:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018927.670580:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333065344/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018927.670590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018927.670593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018927.670595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018927.670598:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3190 00000100:00100000:1.0:1734018927.670600:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333065088:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018927.670601:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018927.670605:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018927.670609:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018927.670618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333065472/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018927.670627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018927.670630:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018927.670632:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018927.670636:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3190 00000100:00080000:0.0:1734018927.670639:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018927.670643:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018927.670647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018927.670656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333065600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018927.670664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018927.670667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018927.670669:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018927.670673:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3190 00000100:00080000:0.0:1734018927.670676:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018927.670679:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018927.670684:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018927.670685:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018927.670692:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333065728/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018927.670700:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018927.670703:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018927.670705:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018927.670708:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3190 00000100:00080000:0.0:1734018927.670711:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018927.670712:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018927/real 1734018927] req@ffff8800a705f800 x1818247333065088/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018927.670715:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018927.670719:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018927.670727:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333065856/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018927.670743:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333065088/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018927.670753:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018927.670757:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018927.670762:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333065088:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018927.670769:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333065216:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018927.670787:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018927.670794:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018927/real 1734018927] req@ffff8800a705d500 x1818247333065216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018927.670805:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333065216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018927.670813:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018927.670815:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018927.670819:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333065216:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018927.670845:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333065344:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018927.670859:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018927.670865:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018927/real 1734018927] req@ffff8800a705d880 x1818247333065344/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018927.670875:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333065344/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018927.670883:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018927.670886:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018927.670889:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333065344:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018927.670895:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333065472:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018927.670908:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018927.670914:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018927/real 1734018927] req@ffff8800a705c380 x1818247333065472/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018927.670924:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333065472/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018927.670932:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018927.670934:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018927.670937:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333065472:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018927.670942:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333065600:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018927.670955:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018927.670961:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018927/real 1734018927] req@ffff8800a705d180 x1818247333065600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018927.670970:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333065600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018927.670978:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018927.670981:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018927.670984:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333065600:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018927.670990:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333065728:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018927.671006:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018927.671012:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018927/real 1734018927] req@ffff8800a705e680 x1818247333065728/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018927.671023:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333065728/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018927.671030:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018927.671032:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018927.671035:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333065728:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018927.671040:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333065856:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018927.671052:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018927.671057:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018927/real 1734018927] req@ffff8800a705c700 x1818247333065856/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018927.671067:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333065856/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018927.671074:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018927.671077:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018927.671079:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333065856:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018928.560509:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018929.562494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018930.564510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018931.566467:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018932.568492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018932.678354:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018932.678364:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018932.678370:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018932.678383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3195 00000100:00080000:0.0:1734018932.678395:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018932.678404:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018932.678452:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333065984/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018932.678489:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018932.678493:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018932.678496:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018932.678500:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3195 00000100:00080000:0.0:1734018932.678504:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018932.678508:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018932.678520:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018932.678532:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333066112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018932.678542:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018932.678546:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018932.678548:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018932.678552:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3195 00000100:00080000:0.0:1734018932.678555:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018932.678559:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018932.678565:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018932.678574:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333066240/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018932.678585:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018932.678588:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018932.678589:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018932.678593:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3195 00000100:00080000:0.0:1734018932.678595:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018932.678599:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018932.678600:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333065984:192.168.201.152@tcp:250:kworker.0 00000008:00100000:0.0:1734018932.678604:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018932.678612:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333066368/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018932.678620:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018932.678623:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018932.678625:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018932.678628:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3195 00000100:00080000:0.0:1734018932.678631:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018932.678634:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018932.678639:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018932.678648:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333066496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018932.678657:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018932.678660:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018932.678662:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018932.678667:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3195 00000100:00080000:0.0:1734018932.678670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018932.678673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018932.678678:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018932.678685:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018932.678687:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333066624/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018932.678695:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018932.678698:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018932.678700:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018932.678704:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3195 00000100:00080000:0.0:1734018932.678706:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018932.678710:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018932.678711:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018932/real 1734018932] req@ffff8800a705df80 x1818247333065984/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018932.678714:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018932.678722:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333066752/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018932.678741:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333065984/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018932.678752:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018932.678756:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018932.678761:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333065984:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018932.678767:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333066112:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018932.678785:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018932.678792:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018932/real 1734018932] req@ffff8800a705ca80 x1818247333066112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018932.678803:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333066112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018932.678811:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018932.678814:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018932.678817:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333066112:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018932.678844:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333066240:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018932.678859:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018932.678865:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018932/real 1734018932] req@ffff8800a705ed80 x1818247333066240/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018932.678875:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333066240/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018932.678883:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018932.678886:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018932.678890:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333066240:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018932.678895:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333066368:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018932.678908:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018932.678913:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018932/real 1734018932] req@ffff8800a705e300 x1818247333066368/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018932.678923:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333066368/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018932.678932:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018932.678934:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018932.678937:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333066368:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018932.678942:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333066496:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018932.678955:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018932.678960:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018932/real 1734018932] req@ffff8800a705c000 x1818247333066496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018932.678970:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333066496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018932.678978:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018932.678980:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018932.678984:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333066496:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018932.678989:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333066624:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018932.679005:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018932.679011:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018932/real 1734018932] req@ffff8800a705dc00 x1818247333066624/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018932.679022:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333066624/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018932.679029:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018932.679031:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018932.679034:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333066624:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018932.679038:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333066752:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018932.679051:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018932.679056:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018932/real 1734018932] req@ffff8800a705f100 x1818247333066752/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018932.679066:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333066752/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018932.679073:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018932.679075:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018932.679078:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333066752:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018933.570489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018934.572464:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018935.574461:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018936.576474:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018937.578502:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018937.686319:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018937.686330:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018937.686336:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018937.686350:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3200 00000100:00080000:0.0:1734018937.686361:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018937.686371:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018937.686419:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333066880/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018937.686464:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018937.686469:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018937.686471:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018937.686476:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3200 00000100:00080000:0.0:1734018937.686479:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018937.686484:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018937.686495:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018937.686508:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333067008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018937.686517:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018937.686521:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018937.686522:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018937.686526:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3200 00000100:00080000:0.0:1734018937.686529:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018937.686533:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018937.686537:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018937.686549:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333067136/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018937.686558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018937.686561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018937.686563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018937.686567:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3200 00000100:00080000:0.0:1734018937.686569:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018937.686572:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333066880:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018937.686574:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018937.686578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018937.686587:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333067264/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018937.686595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018937.686599:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018937.686600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018937.686604:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3200 00000100:00080000:0.0:1734018937.686607:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018937.686611:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018937.686615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018937.686623:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333067392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018937.686632:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018937.686635:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018937.686636:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018937.686640:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3200 00000100:00080000:0.0:1734018937.686643:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018937.686646:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018937.686651:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018937.686659:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333067520/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018937.686659:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018937.686667:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018937.686671:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018937.686673:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018937.686677:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3200 00000100:00080000:0.0:1734018937.686680:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018937.686683:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018937.686686:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018937/real 1734018937] req@ffff8800a705fb80 x1818247333066880/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018937.686687:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018937.686696:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333067648/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018937.686716:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333066880/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018937.686727:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018937.686731:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018937.686736:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333066880:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018937.686743:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333067008:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018937.686761:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018937.686768:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018937/real 1734018937] req@ffff8800a705ea00 x1818247333067008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018937.686779:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333067008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018937.686787:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018937.686789:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018937.686792:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333067008:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018937.686819:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333067136:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018937.686833:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018937.686839:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018937/real 1734018937] req@ffff8800a705f100 x1818247333067136/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018937.686849:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333067136/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018937.686857:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018937.686860:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018937.686863:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333067136:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018937.686869:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333067264:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018937.686881:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018937.686887:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018937/real 1734018937] req@ffff8800a705dc00 x1818247333067264/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018937.686897:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333067264/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018937.686905:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018937.686908:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018937.686911:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333067264:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018937.686915:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333067392:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018937.686928:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018937.686934:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018937/real 1734018937] req@ffff8800a705c000 x1818247333067392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018937.686943:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333067392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018937.686951:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018937.686953:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018937.686957:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333067392:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018937.686962:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333067520:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018937.686978:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018937.686985:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018937/real 1734018937] req@ffff8800a705e300 x1818247333067520/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018937.686995:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333067520/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018937.687002:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018937.687004:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018937.687007:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333067520:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018937.687012:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333067648:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018937.687024:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018937.687030:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018937/real 1734018937] req@ffff8800a705ed80 x1818247333067648/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018937.687039:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333067648/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018937.687093:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018937.687095:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018937.687099:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333067648:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018938.580465:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018939.582436:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018940.584494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018941.586439:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018942.588448:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018942.694343:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018942.694352:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018942.694356:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018942.694367:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3205 00000100:00080000:0.0:1734018942.694377:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018942.694385:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018942.694420:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333067776/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018942.694455:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018942.694460:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018942.694462:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018942.694467:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3205 00000100:00080000:0.0:1734018942.694470:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018942.694474:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018942.694484:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018942.694495:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333067904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018942.694506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018942.694509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018942.694511:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018942.694514:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3205 00000100:00080000:0.0:1734018942.694517:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018942.694520:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018942.694525:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018942.694534:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333068032/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018942.694543:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018942.694546:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018942.694548:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018942.694551:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3205 00000100:00080000:0.0:1734018942.694553:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018942.694557:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018942.694561:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018942.694568:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333068160/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018942.694578:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018942.694581:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018942.694583:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018942.694587:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3205 00000100:00080000:0.0:1734018942.694590:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018942.694593:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018942.694598:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734018942.694598:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333067776:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018942.694606:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333068288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018942.694614:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018942.694617:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018942.694618:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018942.694622:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3205 00000100:00080000:0.0:1734018942.694626:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018942.694629:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018942.694633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018942.694641:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333068416/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018942.694649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018942.694652:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018942.694653:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018942.694657:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3205 00000100:00080000:0.0:1734018942.694675:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018942.694678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018942.694682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018942.694692:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333068544/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018942.694693:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018942.694724:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018942/real 1734018942] req@ffff8800a705ca80 x1818247333067776/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018942.694755:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333067776/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018942.694768:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018942.694772:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018942.694777:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333067776:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018942.694785:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333067904:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018942.694804:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018942.694811:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018942/real 1734018942] req@ffff8800a705df80 x1818247333067904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018942.694822:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333067904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018942.694830:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018942.694833:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018942.694837:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333067904:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018942.694843:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333068032:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018942.694858:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018942.694864:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018942/real 1734018942] req@ffff8800a705c700 x1818247333068032/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018942.694874:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333068032/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018942.694882:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018942.694886:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018942.694889:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333068032:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018942.694919:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333068160:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018942.694934:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018942.694940:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018942/real 1734018942] req@ffff8800a705e680 x1818247333068160/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018942.694951:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333068160/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018942.694994:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018942.694997:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018942.695000:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333068160:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018942.695007:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333068288:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018942.695025:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018942.695032:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018942/real 1734018942] req@ffff8800a705d180 x1818247333068288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018942.695044:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333068288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018942.695052:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018942.695054:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018942.695057:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333068288:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018942.695063:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333068416:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018942.695079:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018942.695086:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018942/real 1734018942] req@ffff8800a705c380 x1818247333068416/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018942.695096:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333068416/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018942.695104:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018942.695106:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018942.695109:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333068416:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018942.695114:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333068544:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018942.695127:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018942.695133:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018942/real 1734018942] req@ffff8800a705d880 x1818247333068544/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018942.695143:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333068544/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018942.695151:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018942.695153:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018942.695156:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333068544:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018943.590463:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018944.592491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018945.594470:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018946.596453:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018947.598474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018947.702360:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018947.702369:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018947.702374:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018947.702385:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3210 00000100:00080000:0.0:1734018947.702395:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018947.702403:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018947.702440:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333068672/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018947.702476:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018947.702480:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018947.702482:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018947.702487:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3210 00000100:00080000:0.0:1734018947.702490:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018947.702494:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018947.702503:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018947.702515:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333068800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018947.702525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018947.702528:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018947.702530:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018947.702534:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3210 00000100:00080000:0.0:1734018947.702536:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018947.702540:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018947.702546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018947.702556:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333068928/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018947.702565:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018947.702569:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018947.702570:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018947.702573:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3210 00000100:00080000:0.0:1734018947.702576:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018947.702579:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018947.702583:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018947.702591:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333069056/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018947.702601:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018947.702604:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018947.702606:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018947.702610:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3210 00000100:00080000:0.0:1734018947.702612:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018947.702612:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333068672:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018947.702616:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018947.702620:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018947.702628:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333069184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018947.702636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018947.702639:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018947.702642:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018947.702646:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3210 00000100:00080000:0.0:1734018947.702649:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018947.702652:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018947.702656:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018947.702665:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333069312/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018947.702672:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018947.702675:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018947.702677:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018947.702680:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3210 00000100:00080000:0.0:1734018947.702693:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018947.702696:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018947.702701:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018947.702707:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018947.702709:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333069440/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018947.702738:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018947/real 1734018947] req@ffff8800a705d500 x1818247333068672/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018947.702768:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333068672/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018947.702781:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018947.702784:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018947.702790:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333068672:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018947.702798:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333068800:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018947.702817:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018947.702824:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018947/real 1734018947] req@ffff8800a705f800 x1818247333068800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018947.702835:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333068800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018947.702843:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018947.702845:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018947.702849:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333068800:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018947.702855:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333068928:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018947.702870:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018947.702928:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018947/real 1734018947] req@ffff8800a705d880 x1818247333068928/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018947.702939:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333068928/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018947.702946:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018947.702950:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018947.702954:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333068928:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018947.702985:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333069056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018947.703003:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018947.703009:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018947/real 1734018947] req@ffff8800a705c380 x1818247333069056/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018947.703020:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333069056/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018947.703028:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018947.703031:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018947.703035:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333069056:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018947.703040:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333069184:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018947.703053:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018947.703059:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018947/real 1734018947] req@ffff8800a705d180 x1818247333069184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018947.703069:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333069184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018947.703076:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018947.703079:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018947.703082:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333069184:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018947.703088:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333069312:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018947.703105:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018947.703111:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018947/real 1734018947] req@ffff8800a705e680 x1818247333069312/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018947.703122:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333069312/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018947.703129:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018947.703131:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018947.703134:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333069312:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018947.703140:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333069440:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018947.703153:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018947.703159:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018947/real 1734018947] req@ffff8800a705c700 x1818247333069440/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018947.703168:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333069440/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018947.703176:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018947.703178:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018947.703181:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333069440:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018948.600487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018949.602498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018950.604407:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018951.606456:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018952.608454:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018952.710328:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018952.710337:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018952.710342:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018952.710352:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3215 00000100:00080000:0.0:1734018952.710362:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018952.710369:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018952.710404:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333069568/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018952.710447:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018952.710452:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018952.710454:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018952.710458:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3215 00000100:00080000:0.0:1734018952.710462:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018952.710465:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018952.710475:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018952.710486:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333069696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018952.710495:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018952.710499:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018952.710500:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018952.710504:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3215 00000100:00080000:0.0:1734018952.710506:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018952.710510:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018952.710514:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018952.710531:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333069824/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018952.710541:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018952.710544:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018952.710545:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018952.710548:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3215 00000100:00080000:0.0:1734018952.710551:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018952.710555:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018952.710560:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018952.710568:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333069952/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018952.710577:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018952.710581:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018952.710582:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018952.710586:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3215 00000100:00100000:1.0:1734018952.710586:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333069568:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018952.710589:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018952.710593:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018952.710597:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018952.710606:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333070080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018952.710614:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018952.710617:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018952.710619:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018952.710623:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3215 00000100:00080000:0.0:1734018952.710625:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018952.710628:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018952.710632:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018952.710641:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333070208/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018952.710649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018952.710653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018952.710654:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018952.710658:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3215 00000100:00080000:0.0:1734018952.710661:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018952.710664:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018952.710668:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018952.710676:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333070336/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018952.710679:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018952.710709:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018952/real 1734018952] req@ffff8800a705df80 x1818247333069568/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018952.710739:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333069568/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018952.710752:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018952.710756:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018952.710761:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333069568:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018952.710769:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333069696:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018952.710820:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018952.710830:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018952/real 1734018952] req@ffff8800a705ca80 x1818247333069696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018952.710842:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333069696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018952.710850:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018952.710853:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018952.710857:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333069696:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018952.710863:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333069824:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018952.710881:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018952.710888:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018952/real 1734018952] req@ffff8800a705ed80 x1818247333069824/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018952.710899:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333069824/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018952.710907:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018952.710911:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018952.710914:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333069824:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018952.710945:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333069952:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018952.710958:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018952.710964:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018952/real 1734018952] req@ffff8800a705e300 x1818247333069952/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018952.710975:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333069952/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018952.710982:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018952.710985:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018952.710988:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333069952:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018952.710993:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333070080:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018952.711006:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018952.711012:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018952/real 1734018952] req@ffff8800a705c000 x1818247333070080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018952.711022:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333070080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018952.711029:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018952.711031:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018952.711034:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333070080:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018952.711041:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333070208:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018952.711057:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018952.711063:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018952/real 1734018952] req@ffff8800a705dc00 x1818247333070208/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018952.711073:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333070208/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018952.711080:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018952.711082:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018952.711085:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333070208:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018952.711091:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333070336:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018952.711104:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018952.711110:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018952/real 1734018952] req@ffff8800a705f100 x1818247333070336/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018952.711119:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333070336/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018952.711127:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018952.711129:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018952.711132:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333070336:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018953.610436:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018954.612488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018955.614503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018956.616505:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018957.618499:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018957.718372:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018957.718383:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018957.718389:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018957.718403:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3220 00000100:00080000:0.0:1734018957.718415:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018957.718425:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018957.718473:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333070464/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018957.718509:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018957.718514:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018957.718516:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018957.718521:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3220 00000100:00080000:0.0:1734018957.718525:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018957.718529:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018957.718541:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018957.718553:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333070592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018957.718565:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018957.718568:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018957.718570:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018957.718574:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3220 00000100:00080000:0.0:1734018957.718576:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018957.718580:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018957.718585:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018957.718597:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333070720/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018957.718606:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018957.718609:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018957.718611:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018957.718614:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3220 00000100:00080000:0.0:1734018957.718617:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018957.718621:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018957.718622:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333070464:192.168.201.152@tcp:250:kworker.0 00000008:00100000:0.0:1734018957.718625:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018957.718634:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333070848/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018957.718643:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018957.718646:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018957.718648:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018957.718651:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3220 00000100:00080000:0.0:1734018957.718655:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018957.718659:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018957.718663:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018957.718672:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333070976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018957.718680:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018957.718683:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018957.718684:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018957.718688:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3220 00000100:00080000:0.0:1734018957.718691:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018957.718694:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018957.718699:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018957.718738:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018957.718739:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333071104/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018957.718748:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018957.718751:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018957.718753:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018957.718756:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3220 00000100:00080000:0.0:1734018957.718759:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018957.718763:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018957.718767:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018957/real 1734018957] req@ffff8800a705ea00 x1818247333070464/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018957.718768:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018957.718777:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333071232/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018957.718797:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333070464/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018957.718809:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018957.718812:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018957.718817:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333070464:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018957.718825:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333070592:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018957.718845:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018957.718852:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018957/real 1734018957] req@ffff8800a705fb80 x1818247333070592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018957.718862:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333070592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018957.718870:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018957.718873:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018957.718876:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333070592:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018957.718903:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333070720:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018957.718917:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018957.718923:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018957/real 1734018957] req@ffff8800a705f100 x1818247333070720/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018957.718933:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333070720/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018957.718941:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018957.718944:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018957.718948:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333070720:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018957.718954:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333070848:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018957.718966:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018957.718972:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018957/real 1734018957] req@ffff8800a705dc00 x1818247333070848/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018957.718981:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333070848/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018957.718989:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018957.718991:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018957.718995:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333070848:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018957.719001:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333070976:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018957.719013:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018957.719019:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018957/real 1734018957] req@ffff8800a705c000 x1818247333070976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018957.719028:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333070976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018957.719036:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018957.719039:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018957.719042:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333070976:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018957.719048:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333071104:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018957.719064:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018957.719070:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018957/real 1734018957] req@ffff8800a705e300 x1818247333071104/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018957.719081:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333071104/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018957.719088:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018957.719090:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018957.719093:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333071104:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018957.719098:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333071232:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018957.719111:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018957.719116:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018957/real 1734018957] req@ffff8800a705ed80 x1818247333071232/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018957.719126:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333071232/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018957.719133:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018957.719135:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018957.719138:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333071232:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018958.620503:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018959.622484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018960.624489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018961.626475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018962.628482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018962.726352:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018962.726363:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018962.726370:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018962.726383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3225 00000100:00080000:0.0:1734018962.726395:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018962.726405:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018962.726455:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333071360/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018962.726492:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018962.726497:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018962.726499:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018962.726504:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3225 00000100:00080000:0.0:1734018962.726507:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018962.726512:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018962.726524:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018962.726536:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333071488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018962.726546:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018962.726550:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018962.726552:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018962.726556:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3225 00000100:00080000:0.0:1734018962.726559:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018962.726562:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018962.726568:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018962.726577:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333071616/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018962.726586:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018962.726589:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018962.726591:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018962.726594:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3225 00000100:00080000:0.0:1734018962.726597:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018962.726601:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018962.726605:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018962.726606:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333071360:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734018962.726614:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333071744/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018962.726658:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018962.726662:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018962.726664:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018962.726667:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3225 00000100:00080000:0.0:1734018962.726670:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018962.726674:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018962.726679:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018962.726690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333071872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018962.726699:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018962.726702:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018962.726705:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018962.726709:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3225 00000100:00080000:0.0:1734018962.726712:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018962.726716:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018962.726720:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018962.726729:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333072000/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018962.726730:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018962.726737:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018962.726740:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018962.726742:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018962.726745:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3225 00000100:00080000:0.0:1734018962.726748:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018962.726751:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018962.726756:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018962.726758:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018962/real 1734018962] req@ffff8800a705ca80 x1818247333071360/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018962.726767:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333072128/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018962.726788:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333071360/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018962.726799:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018962.726802:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018962.726807:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333071360:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018962.726815:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333071488:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018962.726834:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018962.726841:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018962/real 1734018962] req@ffff8800a705df80 x1818247333071488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018962.726852:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333071488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018962.726860:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018962.726862:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018962.726866:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333071488:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018962.726871:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333071616:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018962.726886:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018962.726892:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018962/real 1734018962] req@ffff8800a705c700 x1818247333071616/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018962.726902:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333071616/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018962.726910:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018962.726914:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018962.726917:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333071616:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018962.726946:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333071744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018962.726960:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018962.726967:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018962/real 1734018962] req@ffff8800a705e680 x1818247333071744/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018962.726977:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333071744/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018962.726985:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018962.726987:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018962.726990:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333071744:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018962.726995:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333071872:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018962.727007:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018962.727013:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018962/real 1734018962] req@ffff8800a705d180 x1818247333071872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018962.727022:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333071872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018962.727030:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018962.727032:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018962.727036:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333071872:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018962.727041:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333072000:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018962.727058:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018962.727064:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018962/real 1734018962] req@ffff8800a705c380 x1818247333072000/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018962.727075:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333072000/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018962.727083:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018962.727085:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018962.727088:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333072000:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018962.727093:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333072128:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018962.727105:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018962.727111:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018962/real 1734018962] req@ffff8800a705d880 x1818247333072128/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018962.727120:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333072128/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018962.727127:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018962.727129:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018962.727132:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333072128:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018963.630365:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018964.632465:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018965.634484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018966.636410:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018967.638483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018967.734369:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018967.734380:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018967.734384:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018967.734396:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3230 00000100:00080000:0.0:1734018967.734407:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018967.734415:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018967.734451:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333072256/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018967.734494:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018967.734498:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018967.734500:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018967.734504:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3230 00000100:00080000:0.0:1734018967.734508:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018967.734512:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018967.734522:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018967.734558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333072384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018967.734568:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018967.734571:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018967.734573:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018967.734577:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3230 00000100:00080000:0.0:1734018967.734580:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018967.734583:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018967.734588:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018967.734599:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333072512/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018967.734608:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018967.734611:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018967.734613:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018967.734616:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3230 00000100:00080000:0.0:1734018967.734619:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018967.734623:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018967.734627:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018967.734636:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333072640/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018967.734643:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333072256:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018967.734646:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018967.734650:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018967.734651:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018967.734655:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3230 00000100:00080000:0.0:1734018967.734657:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018967.734661:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018967.734665:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018967.734673:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333072768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018967.734682:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018967.734685:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018967.734687:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018967.734690:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3230 00000100:00080000:0.0:1734018967.734693:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018967.734696:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018967.734700:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018967.734708:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333072896/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018967.734716:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018967.734719:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018967.734721:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018967.734724:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3230 00000100:00080000:0.0:1734018967.734728:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018967.734731:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018967.734735:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018967.734742:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333073024/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018967.734775:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333072384:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018967.734800:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333072512:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018967.734829:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333072640:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018967.734849:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333072768:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018967.734866:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333072896:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018967.734882:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333073024:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018968.640497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018969.642542:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018970.644527:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734018970.740726:0:1956:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734018970.740792:0:1956:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734018970.740817:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018970.740867:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018970.740876:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018970.740884:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018970.740892:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018970.740901:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734018970.740914:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734018970.740964:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018967/real 1734018970] req@ffff8800a705c380 x1818247333072640/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018970.741009:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333072640/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018970.741030:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018970.741036:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018970.741044:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333072640:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018970.741057:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018967/real 1734018970] req@ffff8800a705d180 x1818247333072768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018970.741078:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333072768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018970.741091:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018970.741095:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018970.741101:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333072768:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018970.741113:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018967/real 1734018970] req@ffff8800a705e680 x1818247333072896/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018970.741131:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333072896/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018970.741144:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018970.741149:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018970.741155:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333072896:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018970.741167:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018967/real 1734018970] req@ffff8800a705c700 x1818247333073024/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018970.741187:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333073024/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018970.741200:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018970.741204:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018970.741210:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333073024:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018970.741288:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018967/real 1734018970] req@ffff8800a705f800 x1818247333072256/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018970.741312:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333072256/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018970.741325:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018970.741329:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018970.741335:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333072256:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018970.741346:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018967/real 1734018970] req@ffff8800a705d500 x1818247333072384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018970.741363:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333072384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018970.741385:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018970.741390:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018970.741395:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333072384:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018970.741406:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018967/real 1734018970] req@ffff8800a705d880 x1818247333072512/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018970.741425:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333072512/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018970.741437:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018970.741442:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018970.741447:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333072512:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018971.646423:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018972.648513:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018972.742366:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018972.742377:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018972.742381:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018972.742393:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3235 00000100:00080000:0.0:1734018972.742404:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018972.742413:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018972.742474:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333073152/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018972.742517:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018972.742521:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018972.742523:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018972.742528:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3235 00000100:00080000:0.0:1734018972.742531:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018972.742535:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018972.742545:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018972.742557:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333073280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018972.742567:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018972.742570:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018972.742572:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018972.742576:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3235 00000100:00080000:0.0:1734018972.742578:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018972.742582:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018972.742587:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018972.742595:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333073408/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018972.742603:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018972.742606:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018972.742608:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018972.742611:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3235 00000100:00080000:0.0:1734018972.742614:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018972.742618:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018972.742623:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018972.742631:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333073536/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018972.742637:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333073152:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018972.742641:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018972.742644:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018972.742645:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018972.742649:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3235 00000100:00080000:0.0:1734018972.742651:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018972.742655:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018972.742659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018972.742666:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333073664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018972.742675:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018972.742677:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018972.742679:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018972.742683:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3235 00000100:00080000:0.0:1734018972.742686:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018972.742689:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018972.742693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018972.742702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333073792/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018972.742710:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018972.742714:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018972.742716:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018972.742719:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3235 00000100:00080000:0.0:1734018972.742722:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018972.742726:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734018972.742729:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734018972.742730:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018972.742738:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333073920/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018972.742759:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018972/real 1734018972] req@ffff8800a705df80 x1818247333073152/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018972.742781:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333073152/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734018997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018972.742794:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018972.742798:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018972.742803:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333073152:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018972.742812:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333073280:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018972.742831:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018972.742839:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018972/real 1734018972] req@ffff8800a705ca80 x1818247333073280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018972.742851:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333073280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018972.742859:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018972.742862:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018972.742865:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333073280:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018972.742872:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333073408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018972.742887:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018972.742893:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018972/real 1734018972] req@ffff8800a705ed80 x1818247333073408/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018972.742903:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333073408/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018972.742910:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018972.742913:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018972.742916:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333073408:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018972.742948:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333073536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018972.742961:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018972.742968:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018972/real 1734018972] req@ffff8800a705e300 x1818247333073536/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018972.742979:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333073536/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018972.742995:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018972.742998:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018972.743001:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333073536:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018972.743007:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333073664:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018972.743020:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018972.743026:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018972/real 1734018972] req@ffff8800a705c000 x1818247333073664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018972.743036:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333073664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734018997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018972.743044:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018972.743047:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018972.743050:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333073664:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018972.743056:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333073792:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018972.743069:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018972.743075:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018972/real 1734018972] req@ffff8800a705dc00 x1818247333073792/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018972.743084:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333073792/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018972.743091:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018972.743094:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018972.743097:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333073792:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018972.743104:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333073920:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018972.743117:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018972.743123:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018972/real 1734018972] req@ffff8800a705f100 x1818247333073920/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018972.743133:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333073920/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734018997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018972.743140:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018972.743142:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018972.743145:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333073920:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018973.650510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018974.652501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018975.654469:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018976.656466:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018977.658463:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018977.750338:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018977.750347:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018977.750352:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018977.750396:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3240 00000100:00080000:0.0:1734018977.750407:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018977.750415:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018977.750452:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333074048/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018977.750485:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018977.750489:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018977.750491:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018977.750496:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3240 00000100:00080000:0.0:1734018977.750499:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018977.750503:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018977.750513:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018977.750524:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333074176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018977.750535:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018977.750538:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018977.750540:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018977.750543:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3240 00000100:00080000:0.0:1734018977.750546:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018977.750549:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018977.750554:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018977.750565:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333074304/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018977.750573:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018977.750576:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018977.750578:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018977.750581:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3240 00000100:00080000:0.0:1734018977.750584:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018977.750587:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018977.750592:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018977.750599:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333074432/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018977.750609:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018977.750612:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018977.750614:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018977.750618:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3240 00000100:00100000:1.0:1734018977.750619:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333074048:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018977.750621:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018977.750624:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018977.750629:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018977.750637:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333074560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018977.750645:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018977.750648:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018977.750650:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018977.750654:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3240 00000100:00080000:0.0:1734018977.750656:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018977.750659:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018977.750664:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018977.750672:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333074688/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018977.750680:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018977.750683:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018977.750684:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018977.750688:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3240 00000100:00080000:0.0:1734018977.750700:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018977.750704:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018977.750708:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018977.750716:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333074816/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018977.750716:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018977.750746:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018977/real 1734018977] req@ffff8800a705fb80 x1818247333074048/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018977.750769:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333074048/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018977.750783:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018977.750787:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018977.750792:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333074048:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018977.750800:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333074176:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018977.750820:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018977.750827:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018977/real 1734018977] req@ffff8800a705ea00 x1818247333074176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018977.750847:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333074176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018977.750855:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018977.750858:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018977.750862:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333074176:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018977.750868:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333074304:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018977.750884:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018977.750890:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018977/real 1734018977] req@ffff8800a705f100 x1818247333074304/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018977.750900:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333074304/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018977.750908:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018977.750910:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018977.750913:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333074304:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018977.750944:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333074432:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018977.750957:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018977.750964:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018977/real 1734018977] req@ffff8800a705dc00 x1818247333074432/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018977.750975:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333074432/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018977.750983:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018977.750987:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018977.750990:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333074432:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018977.750997:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333074560:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018977.751010:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018977.751016:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018977/real 1734018977] req@ffff8800a705c000 x1818247333074560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018977.751026:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333074560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018977.751035:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018977.751037:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018977.751040:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333074560:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018977.751046:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333074688:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018977.751059:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018977.751065:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018977/real 1734018977] req@ffff8800a705e300 x1818247333074688/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018977.751075:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333074688/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018977.751082:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018977.751084:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018977.751087:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333074688:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018977.751094:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333074816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018977.751110:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018977.751117:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018977/real 1734018977] req@ffff8800a705ed80 x1818247333074816/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018977.751127:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333074816/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018977.751134:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018977.751137:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018977.751140:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333074816:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018978.660522:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018979.662526:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018980.664507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018981.666517:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018982.668515:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018982.758365:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018982.758375:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018982.758380:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018982.758390:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3245 00000100:00080000:0.0:1734018982.758400:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018982.758408:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018982.758445:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333074944/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018982.758479:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018982.758484:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018982.758486:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018982.758491:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3245 00000100:00080000:0.0:1734018982.758494:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018982.758498:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018982.758508:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018982.758519:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333075072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018982.758529:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018982.758532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018982.758534:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018982.758537:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3245 00000100:00080000:0.0:1734018982.758540:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018982.758543:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018982.758549:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018982.758558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333075200/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018982.758567:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018982.758570:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018982.758572:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018982.758575:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3245 00000100:00080000:0.0:1734018982.758577:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018982.758580:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018982.758585:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018982.758592:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333075328/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018982.758602:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018982.758605:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018982.758606:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734018982.758608:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333074944:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018982.758610:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3245 00000100:00080000:0.0:1734018982.758613:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018982.758618:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018982.758624:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018982.758637:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333075456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018982.758651:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018982.758656:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018982.758659:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018982.758665:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3245 00000100:00080000:0.0:1734018982.758685:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018982.758691:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018982.758699:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018982.758701:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018982.758715:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333075584/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018982.758729:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734018982.758731:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018982/real 1734018982] req@ffff8800a705ca80 x1818247333074944/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734018982.758734:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018982.758737:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018982.758743:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3245 00000100:00080000:0.0:1734018982.758748:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018982.758753:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018982.758754:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333074944/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734018982.758760:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734018982.758768:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018982.758771:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734018982.758773:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333075712/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018982.758777:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333074944:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018982.758785:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333075072:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018982.758805:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018982.758812:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018982/real 1734018982] req@ffff8800a705df80 x1818247333075072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018982.758824:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333075072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018982.758833:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018982.758836:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018982.758840:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333075072:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018982.758846:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333075200:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018982.758862:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018982.758868:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018982/real 1734018982] req@ffff8800a705d880 x1818247333075200/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018982.758878:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333075200/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018982.758886:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018982.758889:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018982.758892:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333075200:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018982.758922:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333075328:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018982.758935:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018982.758941:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018982/real 1734018982] req@ffff8800a705d500 x1818247333075328/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018982.758952:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333075328/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018982.758960:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018982.758962:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018982.758965:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333075328:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018982.758972:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333075456:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018982.758985:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018982.758991:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018982/real 1734018982] req@ffff8800a705f800 x1818247333075456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018982.759002:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333075456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018982.759011:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018982.759013:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018982.759016:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333075456:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018982.759022:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333075584:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018982.759035:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018982.759040:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018982/real 1734018982] req@ffff8800a705c700 x1818247333075584/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018982.759050:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333075584/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018982.759057:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018982.759060:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018982.759063:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333075584:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018982.759069:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333075712:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018982.759082:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018982.759088:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018982/real 1734018982] req@ffff8800a705e680 x1818247333075712/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018982.759099:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333075712/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018982.759107:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018982.759109:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018982.759112:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333075712:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018983.670489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018984.672474:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018985.674484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018986.676491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018987.678494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018987.766361:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018987.766371:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018987.766377:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018987.766390:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3250 00000100:00080000:0.0:1734018987.766402:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018987.766412:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018987.766460:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333075840/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018987.766506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018987.766511:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018987.766513:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018987.766518:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3250 00000100:00080000:0.0:1734018987.766522:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018987.766526:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018987.766538:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018987.766550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333075968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018987.766560:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018987.766564:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018987.766565:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018987.766569:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3250 00000100:00080000:0.0:1734018987.766572:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018987.766576:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018987.766581:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018987.766593:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333076096/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018987.766602:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018987.766605:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018987.766607:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018987.766610:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3250 00000100:00080000:0.0:1734018987.766613:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018987.766617:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734018987.766619:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333075840:192.168.201.152@tcp:250:kworker.0 00000008:00100000:0.0:1734018987.766623:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018987.766631:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333076224/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018987.766639:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018987.766642:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018987.766644:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018987.766647:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3250 00000100:00080000:0.0:1734018987.766650:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018987.766653:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018987.766658:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018987.766666:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333076352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018987.766674:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018987.766678:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018987.766679:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018987.766683:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3250 00000100:00080000:0.0:1734018987.766686:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018987.766689:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018987.766694:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018987.766701:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333076480/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018987.766704:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734018987.766710:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018987.766714:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018987.766715:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018987.766719:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3250 00000100:00080000:0.0:1734018987.766722:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018987.766726:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018987.766730:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734018987.766731:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018987/real 1734018987] req@ffff8800a705d180 x1818247333075840/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734018987.766739:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333076608/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018987.766753:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333075840/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018987.766766:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018987.766770:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018987.766775:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333075840:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018987.766782:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333075968:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018987.766800:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018987.766807:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018987/real 1734018987] req@ffff8800a705c380 x1818247333075968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018987.766818:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333075968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018987.766826:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018987.766830:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018987.766834:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333075968:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018987.766860:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333076096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018987.766875:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018987.766881:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018987/real 1734018987] req@ffff8800a705e680 x1818247333076096/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018987.766892:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333076096/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018987.766900:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018987.766902:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018987.766905:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333076096:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018987.766910:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333076224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018987.766922:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018987.766928:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018987/real 1734018987] req@ffff8800a705c700 x1818247333076224/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018987.766937:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333076224/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018987.766945:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018987.766947:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018987.766950:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333076224:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018987.766955:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333076352:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018987.766972:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018987.766979:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018987/real 1734018987] req@ffff8800a705f800 x1818247333076352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018987.766989:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333076352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018987.766997:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018987.766999:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018987.767002:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333076352:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018987.767007:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333076480:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018987.767020:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018987.767026:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018987/real 1734018987] req@ffff8800a705d500 x1818247333076480/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018987.767036:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333076480/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018987.767043:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018987.767045:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018987.767049:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333076480:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018987.767054:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333076608:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018987.767066:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018987.767072:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018987/real 1734018987] req@ffff8800a705d880 x1818247333076608/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018987.767089:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333076608/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018987.767097:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018987.767099:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018987.767102:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333076608:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018988.680482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018989.682469:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018990.684493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018991.686487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018992.688441:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018992.774327:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018992.774336:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018992.774340:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018992.774351:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3255 00000100:00080000:0.0:1734018992.774361:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018992.774369:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018992.774404:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333076736/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018992.774440:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018992.774444:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018992.774446:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018992.774451:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3255 00000100:00080000:0.0:1734018992.774454:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018992.774458:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018992.774467:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018992.774478:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333076864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018992.774489:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018992.774493:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018992.774494:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018992.774498:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3255 00000100:00080000:0.0:1734018992.774500:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018992.774504:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018992.774508:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018992.774516:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333076992/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018992.774525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018992.774528:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018992.774530:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018992.774533:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3255 00000100:00080000:0.0:1734018992.774535:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018992.774539:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018992.774543:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018992.774550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333077120/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018992.774560:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018992.774563:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018992.774565:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018992.774569:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3255 00000100:00080000:0.0:1734018992.774572:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018992.774572:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333076736:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018992.774575:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018992.774580:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018992.774588:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333077248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018992.774596:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018992.774599:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018992.774601:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018992.774604:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3255 00000100:00080000:0.0:1734018992.774607:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018992.774610:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018992.774615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018992.774624:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333077376/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018992.774631:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018992.774635:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018992.774636:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018992.774639:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3255 00000100:00080000:0.0:1734018992.774652:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018992.774655:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018992.774660:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018992.774668:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333077504/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734018992.774669:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018992.774697:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018992/real 1734018992] req@ffff8800a705df80 x1818247333076736/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018992.774719:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333076736/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018992.774732:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018992.774736:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018992.774741:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333076736:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018992.774749:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333076864:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018992.774769:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018992.774776:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018992/real 1734018992] req@ffff8800a705ca80 x1818247333076864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018992.774787:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333076864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018992.774795:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018992.774800:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018992.774803:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333076864:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018992.774811:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333076992:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018992.774826:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018992.774832:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018992/real 1734018992] req@ffff8800a705ed80 x1818247333076992/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018992.774842:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333076992/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018992.774851:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018992.774853:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018992.774857:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333076992:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018992.774888:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333077120:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018992.774901:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018992.774907:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018992/real 1734018992] req@ffff8800a705e300 x1818247333077120/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018992.774918:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333077120/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018992.774927:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018992.774929:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018992.774932:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333077120:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018992.774938:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333077248:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018992.774954:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018992.775024:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018992/real 1734018992] req@ffff8800a705c000 x1818247333077248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018992.775037:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333077248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018992.775045:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018992.775048:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018992.775052:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333077248:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018992.775058:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333077376:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018992.775076:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018992.775083:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018992/real 1734018992] req@ffff8800a705dc00 x1818247333077376/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018992.775093:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333077376/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018992.775101:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018992.775103:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018992.775147:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333077376:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018992.775154:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333077504:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018992.775171:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018992.775178:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018992/real 1734018992] req@ffff8800a705f100 x1818247333077504/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018992.775196:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333077504/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018992.775204:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018992.775207:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018992.775210:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333077504:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018993.690479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018994.692466:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018995.694460:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018996.696444:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018997.698453:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734018997.782347:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018997.782357:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734018997.782362:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018997.782372:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3260 00000100:00080000:0.0:1734018997.782383:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018997.782390:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734018997.782428:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333077632/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018997.782463:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018997.782467:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018997.782470:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018997.782474:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3260 00000100:00080000:0.0:1734018997.782477:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018997.782481:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018997.782491:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018997.782502:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333077760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018997.782512:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018997.782515:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018997.782517:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018997.782520:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3260 00000100:00080000:0.0:1734018997.782523:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018997.782526:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018997.782532:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734018997.782542:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333077888/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018997.782551:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018997.782555:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018997.782556:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018997.782560:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3260 00000100:00080000:0.0:1734018997.782562:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018997.782565:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018997.782570:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018997.782577:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333078016/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018997.782587:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018997.782590:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734018997.782592:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018997.782596:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3260 00000100:00080000:0.0:1734018997.782598:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734018997.782599:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333077632:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734018997.782602:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018997.782606:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734018997.782614:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333078144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018997.782623:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018997.782626:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734018997.782628:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018997.782633:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3260 00000100:00080000:0.0:1734018997.782635:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018997.782638:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018997.782643:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734018997.782651:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333078272/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734018997.782659:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734018997.782662:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734018997.782663:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734018997.782666:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3260 00000100:00080000:0.0:1734018997.782679:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734018997.782683:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734018997.782687:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734018997.782695:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734018997.782696:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333078400/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734018997.782724:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018997/real 1734018997] req@ffff8800a705ea00 x1818247333077632/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018997.782747:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333077632/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018997.782760:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018997.782764:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018997.782770:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333077632:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734018997.782778:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333077760:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018997.782797:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018997.782804:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018997/real 1734018997] req@ffff8800a705fb80 x1818247333077760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018997.782815:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333077760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018997.782824:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018997.782828:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018997.782832:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333077760:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018997.782839:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333077888:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018997.782855:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018997.782861:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018997/real 1734018997] req@ffff8800a705f100 x1818247333077888/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018997.782870:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333077888/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018997.782878:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018997.782881:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018997.782885:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333077888:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018997.782915:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333078016:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018997.782928:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018997.782934:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018997/real 1734018997] req@ffff8800a705dc00 x1818247333078016/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018997.782945:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333078016/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018997.782953:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018997.782955:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018997.782958:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333078016:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018997.782964:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333078144:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734018997.782980:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018997.782987:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018997/real 1734018997] req@ffff8800a705c000 x1818247333078144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018997.782997:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333078144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018997.783006:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018997.783008:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018997.783011:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333078144:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734018997.783017:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333078272:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018997.783064:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018997.783072:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018997/real 1734018997] req@ffff8800a705e300 x1818247333078272/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018997.783082:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333078272/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018997.783089:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018997.783091:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018997.783094:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333078272:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734018997.783100:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333078400:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734018997.783114:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734018997.783119:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734018997/real 1734018997] req@ffff8800a705ed80 x1818247333078400/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734018997.783137:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333078400/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734018997.783145:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734018997.783147:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734018997.783150:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333078400:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734018998.700432:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734018999.702513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019000.704490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019001.706502:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019002.708463:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019002.790337:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019002.790346:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019002.790351:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019002.790361:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3265 00000100:00080000:0.0:1734019002.790371:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019002.790379:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019002.790415:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333078528/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019002.790459:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019002.790463:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019002.790465:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019002.790470:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3265 00000100:00080000:0.0:1734019002.790474:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019002.790478:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019002.790487:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019002.790498:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333078656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019002.790508:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019002.790511:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019002.790512:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019002.790516:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3265 00000100:00080000:0.0:1734019002.790519:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019002.790522:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019002.790527:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019002.790535:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333078784/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019002.790543:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019002.790547:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019002.790549:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019002.790552:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3265 00000100:00080000:0.0:1734019002.790554:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019002.790559:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019002.790563:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019002.790570:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333078912/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019002.790580:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019002.790584:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019002.790585:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019002.790589:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3265 00000100:00080000:0.0:1734019002.790591:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019002.790595:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019002.790599:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734019002.790604:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333078528:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019002.790607:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333079040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019002.790615:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019002.790618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019002.790619:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019002.790623:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3265 00000100:00080000:0.0:1734019002.790625:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019002.790629:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019002.790633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019002.790641:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333079168/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019002.790649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019002.790653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019002.790654:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019002.790658:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3265 00000100:00080000:0.0:1734019002.790661:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019002.790664:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019002.790668:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019002.790677:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333079296/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019002.790698:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019002.790728:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019002/real 1734019002] req@ffff8800a705ca80 x1818247333078528/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019002.790749:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333078528/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019002.790762:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019002.790766:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019002.790771:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333078528:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019002.790779:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333078656:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019002.790798:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019002.790805:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019002/real 1734019002] req@ffff8800a705df80 x1818247333078656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019002.790816:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333078656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019002.790824:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019002.790828:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019002.790832:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333078656:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019002.790839:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333078784:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019002.790854:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019002.790860:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019002/real 1734019002] req@ffff8800a705d880 x1818247333078784/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019002.790870:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333078784/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019002.790879:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019002.790881:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019002.790885:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333078784:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019002.790914:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333078912:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019002.790927:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019002.790933:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019002/real 1734019002] req@ffff8800a705d500 x1818247333078912/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019002.790991:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333078912/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019002.791001:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019002.791003:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019002.791007:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333078912:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019002.791013:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333079040:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019002.791034:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019002.791041:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019002/real 1734019002] req@ffff8800a705f800 x1818247333079040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019002.791052:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333079040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019002.791060:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019002.791062:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019002.791065:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333079040:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019002.791071:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333079168:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019002.791084:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019002.791091:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019002/real 1734019002] req@ffff8800a705c700 x1818247333079168/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019002.791101:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333079168/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019002.791109:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019002.791111:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019002.791114:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333079168:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019002.791119:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333079296:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019002.791132:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019002.791138:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019002/real 1734019002] req@ffff8800a705e680 x1818247333079296/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019002.791155:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333079296/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019002.791163:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019002.791165:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019002.791168:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333079296:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019003.710471:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019004.712450:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019005.714447:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019006.716493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019007.718460:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019007.798344:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019007.798354:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019007.798359:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019007.798374:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3270 00000100:00080000:0.0:1734019007.798386:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019007.798395:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019007.798444:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333079424/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019007.798483:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019007.798488:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019007.798490:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019007.798495:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3270 00000100:00080000:0.0:1734019007.798499:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019007.798503:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019007.798514:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019007.798526:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333079552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019007.798546:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019007.798550:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019007.798552:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019007.798556:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3270 00000100:00080000:0.0:1734019007.798559:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019007.798562:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019007.798568:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019007.798579:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333079680/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019007.798589:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019007.798592:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019007.798594:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019007.798597:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3270 00000100:00080000:0.0:1734019007.798600:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019007.798604:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734019007.798605:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333079424:192.168.201.152@tcp:250:kworker.0 00000008:00100000:0.0:1734019007.798609:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019007.798616:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333079808/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019007.798624:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019007.798628:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019007.798629:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019007.798633:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3270 00000100:00080000:0.0:1734019007.798637:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019007.798640:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019007.798645:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019007.798653:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333079936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019007.798662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019007.798665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019007.798666:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019007.798670:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3270 00000100:00080000:0.0:1734019007.798673:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019007.798677:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019007.798681:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019007.798689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333080064/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019007.798692:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019007.798697:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019007.798701:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019007.798702:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019007.798706:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3270 00000100:00080000:0.0:1734019007.798709:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019007.798712:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019007.798716:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019007.798718:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019007/real 1734019007] req@ffff8800a705c380 x1818247333079424/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019007.798725:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333080192/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019007.798740:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333079424/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019007.798752:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019007.798755:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019007.798760:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333079424:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019007.798768:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333079552:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019007.798786:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019007.798792:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019007/real 1734019007] req@ffff8800a705d180 x1818247333079552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019007.798804:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333079552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019007.798812:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019007.798815:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019007.798819:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333079552:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019007.798846:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333079680:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019007.798896:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019007.798904:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019007/real 1734019007] req@ffff8800a705e680 x1818247333079680/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019007.798916:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333079680/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019007.798924:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019007.798926:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019007.798929:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333079680:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019007.798935:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333079808:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019007.798947:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019007.798953:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019007/real 1734019007] req@ffff8800a705c700 x1818247333079808/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019007.798962:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333079808/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019007.798970:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019007.798973:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019007.798976:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333079808:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019007.798982:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333079936:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019007.798998:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019007.799004:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019007/real 1734019007] req@ffff8800a705f800 x1818247333079936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019007.799015:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333079936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019007.799023:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019007.799026:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019007.799028:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333079936:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019007.799033:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333080064:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019007.799046:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019007.799051:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019007/real 1734019007] req@ffff8800a705d500 x1818247333080064/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019007.799061:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333080064/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019007.799068:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019007.799070:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019007.799074:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333080064:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019007.799079:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333080192:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019007.799092:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019007.799098:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019007/real 1734019007] req@ffff8800a705d880 x1818247333080192/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019007.799114:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333080192/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019007.799123:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019007.799125:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019007.799128:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333080192:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019008.720498:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019009.722482:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019010.724415:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019011.726444:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019012.728467:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019012.806336:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019012.806345:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019012.806349:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019012.806360:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3275 00000100:00080000:0.0:1734019012.806371:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019012.806378:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019012.806414:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333080320/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019012.806447:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019012.806452:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019012.806454:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019012.806458:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3275 00000100:00080000:0.0:1734019012.806461:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019012.806465:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019012.806475:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019012.806486:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333080448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019012.806496:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019012.806499:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019012.806501:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019012.806505:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3275 00000100:00080000:0.0:1734019012.806508:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019012.806511:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019012.806517:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019012.806525:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333080576/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019012.806534:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019012.806538:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019012.806539:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019012.806542:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3275 00000100:00080000:0.0:1734019012.806545:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019012.806548:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019012.806553:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019012.806560:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333080704/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019012.806569:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019012.806572:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019012.806574:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019012.806578:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3275 00000100:00080000:0.0:1734019012.806580:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019012.806583:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734019012.806586:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333080320:192.168.201.152@tcp:250:kworker.0 00000008:00100000:0.0:1734019012.806588:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019012.806595:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333080832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019012.806604:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019012.806607:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019012.806609:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019012.806613:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3275 00000100:00080000:0.0:1734019012.806616:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019012.806619:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019012.806624:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019012.806633:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333080960/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019012.806641:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019012.806644:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019012.806645:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019012.806649:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3275 00000100:00080000:0.0:1734019012.806662:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019012.806665:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019012.806670:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019012.806678:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333081088/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019012.806679:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019012.806709:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019012/real 1734019012] req@ffff8800a705df80 x1818247333080320/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019012.806730:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333080320/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019012.806743:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019012.806747:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019012.806752:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333080320:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019012.806760:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333080448:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019012.806837:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019012.806845:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019012/real 1734019012] req@ffff8800a705ca80 x1818247333080448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019012.806857:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333080448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019012.806866:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019012.806870:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019012.806873:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333080448:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019012.806880:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333080576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019012.806895:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019012.806901:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019012/real 1734019012] req@ffff8800a705ed80 x1818247333080576/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019012.806912:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333080576/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019012.806920:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019012.806922:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019012.806925:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333080576:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019012.806955:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333080704:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019012.806967:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019012.806973:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019012/real 1734019012] req@ffff8800a705e300 x1818247333080704/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019012.806983:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333080704/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019012.806991:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019012.806994:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019012.806997:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333080704:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019012.807003:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333080832:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019012.807019:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019012.807025:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019012/real 1734019012] req@ffff8800a705c000 x1818247333080832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019012.807036:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333080832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019012.807043:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019012.807046:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019012.807049:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333080832:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019012.807054:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333080960:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019012.807067:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019012.807072:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019012/real 1734019012] req@ffff8800a705dc00 x1818247333080960/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019012.807082:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333080960/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019012.807089:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019012.807091:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019012.807094:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333080960:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019012.807100:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333081088:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019012.807113:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019012.807119:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019012/real 1734019012] req@ffff8800a705f100 x1818247333081088/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019012.807136:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333081088/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019012.807144:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019012.807146:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019012.807149:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333081088:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019013.730472:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019014.732505:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019015.734516:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019016.736502:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019017.738491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019017.814360:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019017.814371:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019017.814375:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019017.814386:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3280 00000100:00080000:0.0:1734019017.814397:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019017.814405:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019017.814442:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333081216/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019017.814485:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019017.814489:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019017.814491:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019017.814496:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3280 00000100:00080000:0.0:1734019017.814500:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019017.814504:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019017.814514:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019017.814525:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333081344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019017.814534:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019017.814538:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019017.814539:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019017.814543:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3280 00000100:00080000:0.0:1734019017.814545:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019017.814549:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019017.814554:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019017.814563:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333081472/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019017.814572:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019017.814575:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019017.814577:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019017.814580:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3280 00000100:00080000:0.0:1734019017.814583:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019017.814587:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019017.814591:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019017.814600:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333081600/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019017.814610:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019017.814613:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019017.814614:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019017.814618:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3280 00000100:00080000:0.0:1734019017.814621:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019017.814624:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333081216:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019017.814625:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019017.814629:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019017.814637:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333081728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019017.814646:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019017.814649:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019017.814650:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019017.814654:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3280 00000100:00080000:0.0:1734019017.814656:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019017.814660:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019017.814664:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019017.814672:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333081856/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019017.814710:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019017.814715:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019017.814717:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019017.814721:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3280 00000100:00080000:0.0:1734019017.814735:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019017.814738:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019017.814743:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019017.814753:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333081984/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019017.814755:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019017.814784:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019017/real 1734019017] req@ffff8800a705fb80 x1818247333081216/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019017.814806:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333081216/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019017.814820:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019017.814823:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019017.814828:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333081216:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019017.814837:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333081344:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019017.814857:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019017.814864:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019017/real 1734019017] req@ffff8800a705ea00 x1818247333081344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019017.814876:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333081344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019017.814884:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019017.814888:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019017.814891:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333081344:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019017.814899:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333081472:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019017.814914:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019017.814921:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019017/real 1734019017] req@ffff8800a705f100 x1818247333081472/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019017.814931:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333081472/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019017.814939:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019017.814942:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019017.814946:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333081472:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019017.814976:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333081600:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019017.814989:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019017.814995:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019017/real 1734019017] req@ffff8800a705dc00 x1818247333081600/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019017.815006:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333081600/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019017.815014:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019017.815017:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019017.815020:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333081600:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019017.815025:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333081728:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019017.815042:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019017.815048:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019017/real 1734019017] req@ffff8800a705c000 x1818247333081728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019017.815059:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333081728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019017.815067:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019017.815070:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019017.815073:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333081728:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019017.815078:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333081856:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019017.815091:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019017.815097:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019017/real 1734019017] req@ffff8800a705e300 x1818247333081856/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019017.815107:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333081856/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019017.815114:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019017.815116:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019017.815119:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333081856:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019017.815125:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333081984:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019017.815139:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019017.815145:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019017/real 1734019017] req@ffff8800a705ed80 x1818247333081984/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019017.815162:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333081984/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019017.815170:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019017.815172:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019017.815175:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333081984:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019018.740474:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019019.742383:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019020.744491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019021.746482:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019022.748481:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019022.822350:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019022.822360:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019022.822364:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019022.822375:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3285 00000100:00080000:0.0:1734019022.822386:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019022.822394:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019022.822429:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333082112/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019022.822463:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019022.822468:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019022.822470:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019022.822474:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3285 00000100:00080000:0.0:1734019022.822477:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019022.822481:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019022.822491:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019022.822502:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333082240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019022.822513:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019022.822516:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019022.822518:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019022.822521:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3285 00000100:00080000:0.0:1734019022.822524:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019022.822527:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019022.822532:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019022.822540:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333082368/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019022.822549:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019022.822552:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019022.822554:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019022.822557:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3285 00000100:00080000:0.0:1734019022.822560:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019022.822563:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019022.822567:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019022.822574:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333082496/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019022.822584:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019022.822587:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019022.822589:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019022.822592:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3285 00000100:00080000:0.0:1734019022.822627:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019022.822632:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734019022.822632:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333082112:192.168.201.152@tcp:250:kworker.0 00000008:00100000:0.0:1734019022.822636:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019022.822646:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333082624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019022.822655:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019022.822658:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019022.822660:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019022.822664:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3285 00000100:00080000:0.0:1734019022.822666:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019022.822670:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019022.822674:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019022.822682:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333082752/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019022.822690:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019022.822693:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019022.822695:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019022.822698:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3285 00000100:00080000:0.0:1734019022.822713:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019022.822717:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019022.822721:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019022.822731:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333082880/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019022.822733:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019022.822762:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019022/real 1734019022] req@ffff8800a705ca80 x1818247333082112/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019022.822784:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333082112/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019022.822797:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019022.822801:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019022.822806:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333082112:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019022.822815:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333082240:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019022.822835:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019022.822841:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019022/real 1734019022] req@ffff8800a705df80 x1818247333082240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019022.822853:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333082240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019022.822861:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019022.822865:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019022.822868:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333082240:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019022.822875:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333082368:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019022.822891:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019022.822897:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019022/real 1734019022] req@ffff8800a705d880 x1818247333082368/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019022.822907:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333082368/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019022.822915:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019022.822918:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019022.822921:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333082368:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019022.822952:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333082496:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019022.822965:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019022.822971:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019022/real 1734019022] req@ffff8800a705d500 x1818247333082496/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019022.822982:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333082496/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019022.822990:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019022.822992:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019022.822995:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333082496:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019022.823001:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333082624:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019022.823017:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019022.823023:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019022/real 1734019022] req@ffff8800a705f800 x1818247333082624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019022.823034:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333082624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019022.823042:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019022.823044:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019022.823047:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333082624:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019022.823053:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333082752:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019022.823065:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019022.823071:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019022/real 1734019022] req@ffff8800a705c700 x1818247333082752/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019022.823081:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333082752/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019022.823089:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019022.823092:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019022.823095:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333082752:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019022.823101:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333082880:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019022.823113:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019022.823119:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019022/real 1734019022] req@ffff8800a705e680 x1818247333082880/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019022.823135:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333082880/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019022.823143:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019022.823146:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019022.823149:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333082880:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019023.750446:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019024.752477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019025.754449:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019026.756506:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019027.758469:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019027.830359:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019027.830370:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019027.830374:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019027.830384:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3290 00000100:00080000:0.0:1734019027.830395:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019027.830403:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019027.830440:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333083008/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019027.830475:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019027.830480:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019027.830482:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019027.830486:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3290 00000100:00080000:0.0:1734019027.830489:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019027.830493:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019027.830504:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019027.830558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333083136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019027.830569:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019027.830572:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019027.830574:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019027.830578:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3290 00000100:00080000:0.0:1734019027.830581:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019027.830585:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019027.830591:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019027.830602:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333083264/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019027.830611:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019027.830614:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019027.830616:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019027.830619:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3290 00000100:00080000:0.0:1734019027.830622:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019027.830625:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019027.830630:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019027.830639:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333083392/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019027.830643:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333083008:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019027.830648:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019027.830652:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019027.830653:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019027.830657:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3290 00000100:00080000:0.0:1734019027.830659:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019027.830662:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019027.830667:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019027.830675:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333083520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019027.830684:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019027.830687:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019027.830689:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019027.830694:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3290 00000100:00080000:0.0:1734019027.830696:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019027.830700:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019027.830704:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019027.830711:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333083648/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019027.830719:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019027.830722:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019027.830724:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019027.830727:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3290 00000100:00080000:0.0:1734019027.830730:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019027.830733:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019027.830738:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019027.830740:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019027.830745:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333083776/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019027.830769:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019027/real 1734019027] req@ffff8800a705d180 x1818247333083008/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019027.830791:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333083008/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019027.830805:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019027.830809:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019027.830814:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333083008:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019027.830822:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333083136:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019027.830842:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019027.830849:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019027/real 1734019027] req@ffff8800a705c380 x1818247333083136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019027.830860:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333083136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019027.830869:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019027.830873:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019027.830877:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333083136:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019027.830884:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333083264:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019027.830899:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019027.830905:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019027/real 1734019027] req@ffff8800a705e680 x1818247333083264/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019027.830915:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333083264/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019027.830923:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019027.830926:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019027.830930:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333083264:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019027.830961:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333083392:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019027.830974:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019027.830980:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019027/real 1734019027] req@ffff8800a705c700 x1818247333083392/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019027.830990:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333083392/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019027.830999:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019027.831001:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019027.831004:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333083392:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019027.831010:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333083520:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019027.831026:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019027.831032:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019027/real 1734019027] req@ffff8800a705f800 x1818247333083520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019027.831043:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333083520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019027.831051:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019027.831053:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019027.831056:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333083520:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019027.831062:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333083648:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019027.831074:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019027.831080:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019027/real 1734019027] req@ffff8800a705d500 x1818247333083648/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019027.831090:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333083648/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019027.831097:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019027.831099:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019027.831102:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333083648:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019027.831108:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333083776:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019027.831121:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019027.831126:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019027/real 1734019027] req@ffff8800a705d880 x1818247333083776/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019027.831143:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333083776/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019027.831151:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019027.831154:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019027.831157:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333083776:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019028.760471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019029.762446:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019030.764521:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019031.766519:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019032.768512:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019032.838326:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019032.838335:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019032.838339:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019032.838350:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3295 00000100:00080000:0.0:1734019032.838360:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019032.838367:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019032.838403:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333083904/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019032.838475:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019032.838480:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019032.838482:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019032.838487:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3295 00000100:00080000:0.0:1734019032.838490:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019032.838494:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019032.838504:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019032.838516:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333084032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019032.838527:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019032.838530:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019032.838531:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019032.838535:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3295 00000100:00080000:0.0:1734019032.838538:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019032.838541:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019032.838546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019032.838554:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333084160/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019032.838562:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019032.838565:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019032.838567:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019032.838570:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3295 00000100:00080000:0.0:1734019032.838573:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019032.838576:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019032.838581:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019032.838588:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333084288/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019032.838597:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019032.838601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019032.838602:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019032.838606:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3295 00000100:00080000:0.0:1734019032.838609:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019032.838609:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333083904:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019032.838612:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019032.838617:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019032.838624:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333084416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019032.838633:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019032.838635:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019032.838637:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019032.838641:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3295 00000100:00080000:0.0:1734019032.838643:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019032.838646:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019032.838651:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019032.838661:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333084544/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019032.838669:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019032.838672:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019032.838674:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019032.838677:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3295 00000100:00080000:0.0:1734019032.838691:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019032.838694:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019032.838699:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019032.838707:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333084672/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019032.838737:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333084032:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019032.838762:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333084160:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019032.838792:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333084288:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019032.838809:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333084416:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019032.838827:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333084544:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019032.838843:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333084672:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019033.770496:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019034.772482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019035.774482:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734019035.844492:0:1955:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:3.0:1734019035.844541:0:1955:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734019035.844561:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734019035.844598:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734019035.844606:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734019035.844613:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734019035.844621:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734019035.844627:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734019035.844633:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734019035.844724:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019032/real 1734019035] req@ffff8800a705e300 x1818247333084288/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019035.844765:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333084288/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019035.844779:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019035.844784:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019035.844791:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333084288:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019035.844800:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019032/real 1734019035] req@ffff8800a705c000 x1818247333084416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019035.844814:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333084416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019035.844822:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019035.844825:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019035.844829:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333084416:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019035.844837:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019032/real 1734019035] req@ffff8800a705dc00 x1818247333084544/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019035.844849:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333084544/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019035.844856:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019035.844859:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019035.844863:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333084544:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019035.844870:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019032/real 1734019035] req@ffff8800a705f100 x1818247333084672/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019035.844881:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333084672/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019035.844888:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019035.844891:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019035.844894:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333084672:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019035.844905:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019032/real 1734019035] req@ffff8800a705df80 x1818247333083904/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019035.844918:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333083904/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019035.844925:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019035.844928:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019035.844931:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333083904:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019035.844938:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019032/real 1734019035] req@ffff8800a705ca80 x1818247333084032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019035.844949:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333084032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019035.844956:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019035.844959:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019035.844963:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333084032:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019035.844969:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019032/real 1734019035] req@ffff8800a705ed80 x1818247333084160/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019035.844980:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333084160/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019035.844987:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019035.844990:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019035.844993:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333084160:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019036.776546:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019037.778500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019037.846330:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019037.846382:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019037.846386:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019037.846398:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3300 00000100:00080000:0.0:1734019037.846409:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019037.846417:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019037.846454:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333084800/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019037.846488:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019037.846492:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019037.846494:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019037.846499:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3300 00000100:00080000:0.0:1734019037.846502:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019037.846506:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019037.846515:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019037.846527:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333084928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019037.846537:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019037.846541:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019037.846542:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019037.846546:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3300 00000100:00080000:0.0:1734019037.846549:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019037.846552:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019037.846557:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019037.846567:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333085056/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019037.846581:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019037.846586:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019037.846588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019037.846593:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3300 00000100:00080000:0.0:1734019037.846597:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019037.846602:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019037.846609:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019037.846622:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333085184/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019037.846637:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019037.846637:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333084800:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019037.846641:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019037.846644:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019037.846649:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3300 00000100:00080000:0.0:1734019037.846654:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019037.846660:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019037.846666:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019037.846680:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333085312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019037.846692:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019037.846697:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019037.846699:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019037.846704:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3300 00000100:00080000:0.0:1734019037.846724:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019037.846730:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019037.846737:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019037.846738:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019037.846751:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333085440/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019037.846764:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019037.846768:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019037/real 1734019037] req@ffff8800a705ea00 x1818247333084800/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019037.846769:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019037.846771:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019037.846777:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3300 00000100:00080000:0.0:1734019037.846781:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019037.846787:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734019037.846792:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333084800/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734019037.846793:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734019037.846805:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1734019037.846807:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333085568/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019037.846809:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019037.846814:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333084800:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019037.846822:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333084928:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019037.846843:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019037.846850:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019037/real 1734019037] req@ffff8800a705fb80 x1818247333084928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019037.846862:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333084928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019037.846870:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019037.846873:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019037.846876:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333084928:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019037.846883:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333085056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019037.846898:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019037.846904:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019037/real 1734019037] req@ffff8800a705ed80 x1818247333085056/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019037.846914:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333085056/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019037.846923:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019037.846926:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019037.846929:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333085056:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019037.846961:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333085184:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019037.846974:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019037.846980:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019037/real 1734019037] req@ffff8800a705ca80 x1818247333085184/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019037.846991:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333085184/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019037.846999:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019037.847001:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019037.847004:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333085184:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019037.847010:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333085312:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019037.847022:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019037.847028:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019037/real 1734019037] req@ffff8800a705df80 x1818247333085312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019037.847038:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333085312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019037.847045:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019037.847048:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019037.847051:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333085312:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019037.847058:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333085440:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019037.847071:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019037.847077:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019037/real 1734019037] req@ffff8800a705f100 x1818247333085440/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019037.847087:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333085440/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019037.847094:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019037.847096:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019037.847099:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333085440:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019037.847105:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333085568:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019037.847118:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019037.847124:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019037/real 1734019037] req@ffff8800a705dc00 x1818247333085568/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019037.847134:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333085568/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019037.847142:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019037.847145:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019037.847148:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333085568:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019038.780533:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019039.782526:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019040.784518:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019041.786512:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019042.788446:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019042.854348:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019042.854358:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019042.854362:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019042.854373:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3305 00000100:00080000:0.0:1734019042.854383:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019042.854391:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019042.854428:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333085696/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019042.854461:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019042.854466:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019042.854468:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019042.854473:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3305 00000100:00080000:0.0:1734019042.854476:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019042.854480:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019042.854490:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019042.854500:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333085824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019042.854510:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019042.854513:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019042.854515:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019042.854519:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3305 00000100:00080000:0.0:1734019042.854521:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019042.854525:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019042.854530:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019042.854538:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333085952/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019042.854548:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019042.854551:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019042.854552:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019042.854556:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3305 00000100:00080000:0.0:1734019042.854558:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019042.854562:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019042.854566:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019042.854574:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333086080/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019042.854583:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019042.854586:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019042.854588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019042.854592:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3305 00000100:00080000:0.0:1734019042.854594:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019042.854596:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333085696:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019042.854597:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019042.854602:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019042.854610:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333086208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019042.854618:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019042.854621:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019042.854624:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019042.854628:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3305 00000100:00080000:0.0:1734019042.854631:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019042.854634:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019042.854638:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019042.854646:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333086336/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019042.854654:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019042.854657:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019042.854659:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019042.854662:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3305 00000100:00080000:0.0:1734019042.854674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019042.854678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019042.854682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019042.854692:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333086464/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019042.854692:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019042.854722:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019042/real 1734019042] req@ffff8800a705c000 x1818247333085696/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019042.854745:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333085696/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019042.854758:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019042.854762:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019042.854767:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333085696:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019042.854775:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333085824:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019042.854795:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019042.854803:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019042/real 1734019042] req@ffff8800a705e300 x1818247333085824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019042.854815:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333085824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019042.854823:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019042.854826:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019042.854829:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333085824:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019042.854836:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333085952:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019042.854852:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019042.854858:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019042/real 1734019042] req@ffff8800a705d880 x1818247333085952/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019042.854868:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333085952/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019042.854875:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019042.854879:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019042.854882:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333085952:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019042.854913:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333086080:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019042.854927:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019042.854933:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019042/real 1734019042] req@ffff8800a705d500 x1818247333086080/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019042.854943:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333086080/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019042.854951:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019042.854953:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019042.854956:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333086080:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019042.854962:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333086208:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019042.854975:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019042.854981:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019042/real 1734019042] req@ffff8800a705f800 x1818247333086208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019042.854991:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333086208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019042.854998:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019042.855001:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019042.855004:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333086208:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019042.855010:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333086336:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019042.855027:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019042.855033:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019042/real 1734019042] req@ffff8800a705c700 x1818247333086336/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019042.855044:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333086336/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019042.855050:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019042.855053:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019042.855056:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333086336:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019042.855061:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333086464:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019042.855074:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019042.855080:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019042/real 1734019042] req@ffff8800a705e680 x1818247333086464/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019042.855090:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333086464/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019042.855097:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019042.855099:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019042.855102:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333086464:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019043.790479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019044.792484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019045.794502:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019046.796509:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019047.798455:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019047.862364:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019047.862374:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019047.862381:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019047.862395:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3310 00000100:00080000:0.0:1734019047.862407:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019047.862416:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019047.862465:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333086592/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019047.862511:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019047.862516:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019047.862518:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019047.862523:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3310 00000100:00080000:0.0:1734019047.862527:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019047.862532:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019047.862544:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019047.862556:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333086720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019047.862566:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019047.862570:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019047.862572:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019047.862576:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3310 00000100:00080000:0.0:1734019047.862579:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019047.862582:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019047.862587:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019047.862598:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333086848/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019047.862607:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019047.862610:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019047.862611:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019047.862615:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3310 00000100:00080000:0.0:1734019047.862617:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019047.862622:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019047.862627:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019047.862628:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333086592:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019047.862636:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333086976/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019047.862645:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019047.862648:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019047.862650:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019047.862653:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3310 00000100:00080000:0.0:1734019047.862656:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019047.862659:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019047.862664:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019047.862672:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333087104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019047.862680:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019047.862683:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019047.862685:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019047.862689:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3310 00000100:00080000:0.0:1734019047.862701:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019047.862705:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019047.862709:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019047.862716:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019047.862718:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333087232/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019047.862727:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019047.862731:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019047.862733:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019047.862737:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3310 00000100:00080000:0.0:1734019047.862740:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019047.862742:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019047/real 1734019047] req@ffff8800a705c380 x1818247333086592/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019047.862744:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019047.862748:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019047.862756:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333087360/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019047.862772:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333086592/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019047.862784:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019047.862788:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019047.862793:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333086592:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019047.862800:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333086720:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019047.862819:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019047.862826:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019047/real 1734019047] req@ffff8800a705d180 x1818247333086720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019047.862837:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333086720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019047.862845:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019047.862848:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019047.862851:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333086720:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019047.862857:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333086848:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019047.862872:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019047.862877:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019047/real 1734019047] req@ffff8800a705e680 x1818247333086848/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019047.862887:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333086848/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019047.862895:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019047.862898:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019047.862901:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333086848:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019047.862930:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333086976:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019047.862943:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019047.862950:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019047/real 1734019047] req@ffff8800a705c700 x1818247333086976/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019047.862961:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333086976/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019047.862968:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019047.862970:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019047.862973:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333086976:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019047.862978:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333087104:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019047.862990:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019047.862996:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019047/real 1734019047] req@ffff8800a705f800 x1818247333087104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019047.863006:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333087104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019047.863013:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019047.863015:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019047.863019:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333087104:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019047.863025:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333087232:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019047.863041:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019047.863047:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019047/real 1734019047] req@ffff8800a705d500 x1818247333087232/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019047.863057:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333087232/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019047.863064:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019047.863066:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019047.863069:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333087232:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019047.863074:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333087360:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019047.863087:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019047.863092:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019047/real 1734019047] req@ffff8800a705d880 x1818247333087360/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019047.863102:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333087360/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019047.863109:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019047.863111:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019047.863114:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333087360:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019048.800454:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019049.802457:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019050.804466:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019051.806466:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019052.808367:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019052.870339:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019052.870349:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019052.870353:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019052.870364:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3315 00000100:00080000:0.0:1734019052.870374:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019052.870382:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019052.870417:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333087488/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019052.870453:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019052.870457:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019052.870459:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019052.870463:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3315 00000100:00080000:0.0:1734019052.870466:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019052.870470:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019052.870480:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019052.870490:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333087616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019052.870501:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019052.870505:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019052.870506:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019052.870510:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3315 00000100:00080000:0.0:1734019052.870512:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019052.870516:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019052.870520:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019052.870528:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333087744/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019052.870538:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019052.870542:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019052.870543:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019052.870547:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3315 00000100:00080000:0.0:1734019052.870549:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019052.870553:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734019052.870556:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333087488:192.168.201.152@tcp:250:kworker.0 00000008:00100000:0.0:1734019052.870557:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019052.870565:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333087872/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019052.870573:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019052.870576:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019052.870578:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019052.870582:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3315 00000100:00080000:0.0:1734019052.870586:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019052.870589:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019052.870593:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019052.870601:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333088000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019052.870609:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019052.870612:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019052.870613:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019052.870617:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3315 00000100:00080000:0.0:1734019052.870620:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019052.870623:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019052.870627:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019052.870635:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019052.870636:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333088128/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019052.870643:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019052.870647:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019052.870648:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019052.870651:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3315 00000100:00080000:0.0:1734019052.870654:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019052.870657:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734019052.870659:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019052/real 1734019052] req@ffff8800a705e300 x1818247333087488/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734019052.870661:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019052.870669:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333088256/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019052.870678:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333087488/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019052.870686:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019052.870689:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019052.870692:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333087488:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019052.870698:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333087616:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019052.870710:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019052.870714:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019052/real 1734019052] req@ffff8800a705c000 x1818247333087616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019052.870720:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333087616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019052.870725:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019052.870726:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019052.870728:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333087616:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019052.870749:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333087744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019052.870758:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019052.870761:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019052/real 1734019052] req@ffff8800a705dc00 x1818247333087744/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019052.870766:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333087744/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019052.870771:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019052.870773:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019052.870775:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333087744:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019052.870779:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333087872:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019052.870786:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019052.870789:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019052/real 1734019052] req@ffff8800a705f100 x1818247333087872/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019052.870795:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333087872/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019052.870799:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019052.870801:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019052.870803:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333087872:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019052.870806:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333088000:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019052.870813:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019052.870816:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019052/real 1734019052] req@ffff8800a705df80 x1818247333088000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019052.870821:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333088000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019052.870826:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019052.870827:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019052.870829:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333088000:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019052.870833:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333088128:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019052.870843:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019052.870846:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019052/real 1734019052] req@ffff8800a705ca80 x1818247333088128/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019052.870852:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333088128/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019052.870855:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019052.870857:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019052.870858:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333088128:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019052.870861:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333088256:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019052.870869:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019052.870872:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019052/real 1734019052] req@ffff8800a705ed80 x1818247333088256/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019052.870877:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333088256/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019052.870880:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019052.870882:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019052.870883:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333088256:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019053.810459:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019054.812447:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019055.814508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019056.816491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019057.818478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019057.878357:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019057.878367:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019057.878371:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019057.878382:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3320 00000100:00080000:0.0:1734019057.878393:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019057.878401:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019057.878437:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333088384/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019057.878472:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019057.878476:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019057.878478:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019057.878483:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3320 00000100:00080000:0.0:1734019057.878485:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019057.878489:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019057.878499:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019057.878510:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333088512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019057.878520:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019057.878523:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019057.878525:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019057.878528:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3320 00000100:00080000:0.0:1734019057.878531:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019057.878534:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019057.878540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019057.878550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333088640/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019057.878559:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019057.878562:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019057.878564:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019057.878567:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3320 00000100:00080000:0.0:1734019057.878569:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019057.878573:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019057.878577:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019057.878585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333088768/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019057.878595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019057.878598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019057.878599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019057.878603:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3320 00000100:00080000:0.0:1734019057.878606:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019057.878609:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734019057.878610:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333088384:192.168.201.152@tcp:250:kworker.0 00000008:00100000:0.0:1734019057.878614:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019057.878622:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333088896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019057.878631:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019057.878634:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019057.878636:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019057.878640:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3320 00000100:00080000:0.0:1734019057.878642:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019057.878646:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019057.878650:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019057.878657:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333089024/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019057.878665:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019057.878668:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019057.878670:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019057.878673:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3320 00000100:00080000:0.0:1734019057.878686:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019057.878689:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019057.878694:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019057.878702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333089152/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019057.878705:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019057.878734:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019057/real 1734019057] req@ffff8800a705fb80 x1818247333088384/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019057.878765:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333088384/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019057.878777:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019057.878781:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019057.878786:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333088384:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019057.878794:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333088512:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019057.878814:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019057.878821:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019057/real 1734019057] req@ffff8800a705ea00 x1818247333088512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019057.878832:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333088512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019057.878840:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019057.878844:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019057.878847:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333088512:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019057.878854:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333088640:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019057.878869:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019057.878874:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019057/real 1734019057] req@ffff8800a705ed80 x1818247333088640/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019057.878885:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333088640/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019057.878892:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019057.878895:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019057.878899:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333088640:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019057.878929:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333088768:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019057.878943:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019057.878950:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019057/real 1734019057] req@ffff8800a705ca80 x1818247333088768/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019057.878961:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333088768/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019057.878968:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019057.878970:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019057.878973:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333088768:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019057.878979:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333088896:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019057.878992:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019057.878998:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019057/real 1734019057] req@ffff8800a705df80 x1818247333088896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019057.879053:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333088896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019057.879061:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019057.879064:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019057.879067:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333088896:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019057.879073:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333089024:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019057.879093:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019057.879101:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019057/real 1734019057] req@ffff8800a705f100 x1818247333089024/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019057.879111:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333089024/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019057.879120:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019057.879122:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019057.879126:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333089024:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019057.879131:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333089152:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019057.879144:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019057.879150:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019057/real 1734019057] req@ffff8800a705dc00 x1818247333089152/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019057.879160:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333089152/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019057.879167:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019057.879170:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019057.879172:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333089152:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019058.820488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019059.822483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019060.824437:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019061.826450:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019062.828502:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019062.886356:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019062.886365:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019062.886369:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019062.886381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3325 00000100:00080000:0.0:1734019062.886392:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019062.886399:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019062.886436:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333089280/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019062.886479:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019062.886483:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019062.886485:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019062.886490:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3325 00000100:00080000:0.0:1734019062.886493:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019062.886497:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019062.886507:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019062.886518:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333089408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019062.886528:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019062.886531:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019062.886533:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019062.886537:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3325 00000100:00080000:0.0:1734019062.886539:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019062.886543:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019062.886548:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019062.886555:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333089536/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019062.886564:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019062.886567:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019062.886569:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019062.886572:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3325 00000100:00080000:0.0:1734019062.886574:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019062.886578:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019062.886582:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019062.886590:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333089664/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019062.886600:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019062.886604:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019062.886605:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019062.886608:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3325 00000100:00080000:0.0:1734019062.886611:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019062.886614:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019062.886619:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734019062.886622:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333089280:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019062.886627:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333089792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019062.886635:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019062.886638:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019062.886640:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019062.886644:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3325 00000100:00080000:0.0:1734019062.886646:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019062.886649:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019062.886653:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019062.886661:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333089920/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019062.886670:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019062.886673:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019062.886675:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019062.886679:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3325 00000100:00080000:0.0:1734019062.886698:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019062.886702:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019062.886706:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019062.886716:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333090048/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019062.886716:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019062.886745:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019062/real 1734019062] req@ffff8800a705c000 x1818247333089280/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019062.886776:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333089280/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019062.886789:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019062.886792:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019062.886798:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333089280:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019062.886805:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333089408:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019062.886824:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019062.886831:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019062/real 1734019062] req@ffff8800a705e300 x1818247333089408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019062.886842:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333089408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019062.886851:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019062.886853:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019062.886857:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333089408:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019062.886863:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333089536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019062.886878:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019062.886884:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019062/real 1734019062] req@ffff8800a705d880 x1818247333089536/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019062.886894:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333089536/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019062.886901:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019062.886905:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019062.886908:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333089536:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019062.886984:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333089664:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019062.887003:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019062.887010:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019062/real 1734019062] req@ffff8800a705d500 x1818247333089664/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019062.887022:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333089664/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019062.887030:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019062.887032:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019062.887035:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333089664:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019062.887041:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333089792:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019062.887054:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019062.887060:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019062/real 1734019062] req@ffff8800a705f800 x1818247333089792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019062.887070:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333089792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019062.887079:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019062.887081:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019062.887085:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333089792:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019062.887090:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333089920:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019062.887107:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019062.887113:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019062/real 1734019062] req@ffff8800a705c700 x1818247333089920/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019062.887124:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333089920/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019062.887130:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019062.887133:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019062.887136:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333089920:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019062.887141:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333090048:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019062.887154:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019062.887160:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019062/real 1734019062] req@ffff8800a705e680 x1818247333090048/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019062.887171:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333090048/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019062.887178:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019062.887181:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019062.887184:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333090048:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019063.830497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019064.832456:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019065.834444:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019066.836473:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019067.838517:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019067.894337:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019067.894349:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019067.894354:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019067.894368:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3330 00000100:00080000:0.0:1734019067.894380:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019067.894390:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019067.894441:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333090176/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019067.894479:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019067.894483:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019067.894486:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019067.894491:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3330 00000100:00080000:0.0:1734019067.894494:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019067.894499:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019067.894511:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019067.894523:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333090304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019067.894535:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019067.894538:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019067.894540:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019067.894544:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3330 00000100:00080000:0.0:1734019067.894547:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019067.894550:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019067.894555:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019067.894567:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333090432/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019067.894576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019067.894580:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019067.894581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019067.894584:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3330 00000100:00080000:0.0:1734019067.894587:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019067.894591:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734019067.894593:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333090176:192.168.201.152@tcp:250:kworker.0 00000008:00100000:0.0:1734019067.894596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019067.894604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333090560/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019067.894612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019067.894616:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019067.894618:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019067.894621:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3330 00000100:00080000:0.0:1734019067.894625:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019067.894629:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019067.894634:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019067.894642:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333090688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019067.894650:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019067.894653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019067.894655:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019067.894659:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3330 00000100:00080000:0.0:1734019067.894662:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019067.894665:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019067.894670:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019067.894678:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333090816/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019067.894679:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019067.894686:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019067.894690:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019067.894691:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019067.894695:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3330 00000100:00080000:0.0:1734019067.894698:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019067.894701:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019067.894705:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019067.894705:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019067/real 1734019067] req@ffff8800a705d180 x1818247333090176/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019067.894714:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333090944/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019067.894736:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333090176/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019067.894748:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019067.894751:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019067.894756:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333090176:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019067.894763:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333090304:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019067.894782:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019067.894788:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019067/real 1734019067] req@ffff8800a705c380 x1818247333090304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019067.894800:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333090304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019067.894807:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019067.894810:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019067.894813:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333090304:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019067.894894:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333090432:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019067.894914:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019067.894921:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019067/real 1734019067] req@ffff8800a705e680 x1818247333090432/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019067.894932:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333090432/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019067.894940:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019067.894944:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019067.894948:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333090432:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019067.894954:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333090560:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019067.894967:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019067.894972:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019067/real 1734019067] req@ffff8800a705c700 x1818247333090560/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019067.894983:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333090560/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019067.894991:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019067.894993:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019067.894996:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333090560:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019067.895001:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333090688:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019067.895014:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019067.895020:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019067/real 1734019067] req@ffff8800a705f800 x1818247333090688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019067.895029:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333090688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019067.895037:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019067.895039:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019067.895043:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333090688:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019067.895048:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333090816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019067.895065:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019067.895071:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019067/real 1734019067] req@ffff8800a705d500 x1818247333090816/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019067.895081:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333090816/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019067.895088:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019067.895090:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019067.895093:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333090816:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019067.895098:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333090944:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019067.895110:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019067.895116:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019067/real 1734019067] req@ffff8800a705d880 x1818247333090944/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019067.895125:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333090944/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019067.895133:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019067.895135:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019067.895138:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333090944:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019068.840489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019069.842508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019070.844680:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019071.846408:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019072.848450:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019072.902365:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019072.902375:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019072.902379:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019072.902389:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3335 00000100:00080000:0.0:1734019072.902399:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019072.902406:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019072.902441:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333091072/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019072.902473:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019072.902478:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019072.902480:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019072.902484:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3335 00000100:00080000:0.0:1734019072.902487:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019072.902491:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019072.902501:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019072.902512:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333091200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019072.902522:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019072.902525:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019072.902527:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019072.902531:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3335 00000100:00080000:0.0:1734019072.902534:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019072.902537:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019072.902542:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019072.902550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333091328/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019072.902559:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019072.902562:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019072.902564:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019072.902567:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3335 00000100:00080000:0.0:1734019072.902570:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019072.902573:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019072.902577:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019072.902583:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333091072:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019072.902585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333091456/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019072.902594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019072.902597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019072.902599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019072.902602:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3335 00000100:00080000:0.0:1734019072.902604:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019072.902608:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019072.902612:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019072.902619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333091584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019072.902628:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019072.902631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019072.902633:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019072.902637:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3335 00000100:00080000:0.0:1734019072.902640:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019072.902643:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019072.902647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019072.902650:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019072.902656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333091712/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019072.902664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019072.902667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019072.902669:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019072.902671:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019072/real 1734019072] req@ffff8800a705e300 x1818247333091072/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019072.902672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3335 00000100:00080000:0.0:1734019072.902675:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019072.902678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019072.902682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019072.902690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333091840/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019072.902697:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333091072/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019072.902708:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019072.902712:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019072.902717:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333091072:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019072.902724:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333091200:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019072.902740:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019072.902803:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019072/real 1734019072] req@ffff8800a705c000 x1818247333091200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019072.902816:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333091200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019072.902824:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019072.902827:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019072.902830:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333091200:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019072.902837:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333091328:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019072.902855:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019072.902861:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019072/real 1734019072] req@ffff8800a705dc00 x1818247333091328/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019072.902872:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333091328/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019072.902879:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019072.902882:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019072.902886:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333091328:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019072.902909:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333091456:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019072.902922:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019072.902928:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019072/real 1734019072] req@ffff8800a705f100 x1818247333091456/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019072.902938:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333091456/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019072.902945:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019072.902947:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019072.902950:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333091456:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019072.902955:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333091584:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019072.902967:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019072.902973:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019072/real 1734019072] req@ffff8800a705df80 x1818247333091584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019072.902983:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333091584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019072.902990:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019072.902992:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019072.902995:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333091584:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019072.903000:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333091712:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019072.903016:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019072.903021:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019072/real 1734019072] req@ffff8800a705ca80 x1818247333091712/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019072.903031:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333091712/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019072.903038:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019072.903040:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019072.903043:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333091712:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019072.903048:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333091840:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019072.903060:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019072.903066:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019072/real 1734019072] req@ffff8800a705ed80 x1818247333091840/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019072.903075:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333091840/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019072.903082:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019072.903085:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019072.903087:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333091840:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019073.850491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019074.852436:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019075.854491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019076.856479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019077.858484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019077.910340:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019077.910349:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019077.910353:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019077.910364:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3340 00000100:00080000:0.0:1734019077.910374:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019077.910382:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019077.910417:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333091968/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019077.910459:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019077.910463:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019077.910465:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019077.910469:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3340 00000100:00080000:0.0:1734019077.910473:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019077.910476:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019077.910486:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019077.910497:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333092096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019077.910506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019077.910509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019077.910511:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019077.910515:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3340 00000100:00080000:0.0:1734019077.910517:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019077.910521:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019077.910525:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019077.910535:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333092224/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019077.910543:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019077.910546:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019077.910548:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019077.910551:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3340 00000100:00080000:0.0:1734019077.910554:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019077.910557:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019077.910561:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019077.910569:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333092352/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019077.910575:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333091968:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019077.910578:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019077.910581:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019077.910583:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019077.910586:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3340 00000100:00080000:0.0:1734019077.910589:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019077.910592:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019077.910596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019077.910604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333092480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019077.910612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019077.910615:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019077.910616:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019077.910620:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3340 00000100:00080000:0.0:1734019077.910632:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019077.910636:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019077.910641:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019077.910644:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019077.910649:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333092608/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019077.910657:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019077.910693:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019077.910695:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019077.910699:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3340 00000100:00080000:0.0:1734019077.910703:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019077.910706:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734019077.910710:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019077/real 1734019077] req@ffff8800a705ea00 x1818247333091968/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734019077.910711:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019077.910721:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333092736/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019077.910737:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333091968/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019077.910748:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019077.910751:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019077.910756:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333091968:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019077.910763:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333092096:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019077.910783:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019077.910790:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019077/real 1734019077] req@ffff8800a705fb80 x1818247333092096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019077.910801:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333092096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019077.910809:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019077.910812:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019077.910815:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333092096:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019077.910821:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333092224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019077.910834:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019077.910840:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019077/real 1734019077] req@ffff8800a705ed80 x1818247333092224/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019077.910850:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333092224/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019077.910857:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019077.910860:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019077.910864:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333092224:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019077.910888:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333092352:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019077.910900:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019077.910906:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019077/real 1734019077] req@ffff8800a705ca80 x1818247333092352/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019077.910917:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333092352/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019077.910924:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019077.910926:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019077.910929:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333092352:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019077.910934:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333092480:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019077.910946:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019077.910951:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019077/real 1734019077] req@ffff8800a705df80 x1818247333092480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019077.910961:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333092480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019077.910968:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019077.910970:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019077.910974:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333092480:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019077.910979:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333092608:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019077.910994:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019077.911000:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019077/real 1734019077] req@ffff8800a705f100 x1818247333092608/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019077.911010:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333092608/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019077.911017:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019077.911019:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019077.911022:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333092608:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019077.911027:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333092736:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019077.911038:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019077.911044:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019077/real 1734019077] req@ffff8800a705dc00 x1818247333092736/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019077.911054:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333092736/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019077.911061:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019077.911063:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019077.911066:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333092736:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019078.860508:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019079.862452:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019080.864515:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019081.866485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019082.868467:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019082.918342:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019082.918352:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019082.918356:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019082.918367:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3345 00000100:00080000:0.0:1734019082.918377:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019082.918385:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019082.918420:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333092864/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019082.918454:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019082.918458:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019082.918460:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019082.918465:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3345 00000100:00080000:0.0:1734019082.918468:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019082.918472:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019082.918481:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019082.918492:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333092992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019082.918503:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019082.918506:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019082.918508:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019082.918511:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3345 00000100:00080000:0.0:1734019082.918514:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019082.918517:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019082.918522:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019082.918530:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333093120/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019082.918538:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019082.918541:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019082.918543:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019082.918546:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3345 00000100:00080000:0.0:1734019082.918549:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019082.918552:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019082.918556:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019082.918564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333093248/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019082.918567:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333092864:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019082.918616:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019082.918621:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019082.918623:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019082.918626:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3345 00000100:00080000:0.0:1734019082.918630:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019082.918633:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019082.918638:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019082.918647:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333093376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019082.918656:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019082.918659:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019082.918661:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019082.918665:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3345 00000100:00080000:0.0:1734019082.918668:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1734019082.918669:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019082.918672:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019082.918676:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019082.918684:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333093504/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019082.918691:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019082/real 1734019082] req@ffff8800a705c000 x1818247333092864/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019082.918692:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019082.918695:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019082.918697:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019082.918700:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3345 00000100:00080000:0.0:1734019082.918703:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019082.918706:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019082.918710:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019082.918718:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333092864/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019082.918720:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333093632/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019082.918727:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019082.918731:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019082.918736:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333092864:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019082.918744:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333092992:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019082.918761:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019082.918767:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019082/real 1734019082] req@ffff8800a705e300 x1818247333092992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019082.918778:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333092992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019082.918786:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019082.918788:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019082.918792:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333092992:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019082.918798:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333093120:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019082.918811:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019082.918817:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019082/real 1734019082] req@ffff8800a705d880 x1818247333093120/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019082.918827:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333093120/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019082.918834:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019082.918838:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019082.918841:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333093120:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019082.918866:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333093248:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019082.918878:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019082.918884:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019082/real 1734019082] req@ffff8800a705d500 x1818247333093248/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019082.918894:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333093248/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019082.918902:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019082.918904:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019082.918907:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333093248:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019082.918912:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333093376:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019082.918923:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019082.918929:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019082/real 1734019082] req@ffff8800a705f800 x1818247333093376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019082.918938:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333093376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019082.918946:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019082.918948:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019082.918951:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333093376:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019082.918956:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333093504:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019082.918970:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019082.918976:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019082/real 1734019082] req@ffff8800a705c700 x1818247333093504/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019082.918987:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333093504/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019082.918993:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019082.918995:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019082.918998:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333093504:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019082.919003:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333093632:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019082.919015:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019082.919020:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019082/real 1734019082] req@ffff8800a705e680 x1818247333093632/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019082.919030:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333093632/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019082.919037:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019082.919039:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019082.919042:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333093632:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019083.870481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019084.872410:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019085.874496:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019086.876492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019087.878469:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019087.926338:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019087.926347:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019087.926352:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019087.926362:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3350 00000100:00080000:0.0:1734019087.926371:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019087.926379:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019087.926415:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333093760/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019087.926450:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019087.926454:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019087.926456:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019087.926461:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3350 00000100:00080000:0.0:1734019087.926464:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019087.926468:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019087.926482:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019087.926526:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333093888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019087.926537:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019087.926540:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019087.926542:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019087.926545:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3350 00000100:00080000:0.0:1734019087.926548:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019087.926551:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019087.926557:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019087.926568:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333094016/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019087.926578:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019087.926581:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019087.926583:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019087.926586:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3350 00000100:00100000:1.0:1734019087.926586:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333093760:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019087.926589:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019087.926592:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019087.926597:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019087.926604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333094144/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019087.926612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019087.926615:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019087.926617:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019087.926620:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3350 00000100:00080000:0.0:1734019087.926623:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019087.926626:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019087.926630:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019087.926638:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333094272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019087.926646:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019087.926649:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019087.926651:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1734019087.926654:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019087.926655:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3350 00000100:00080000:0.0:1734019087.926658:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019087.926661:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019087.926665:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019087.926673:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333094400/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019087.926677:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019087/real 1734019087] req@ffff8800a705c380 x1818247333093760/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019087.926680:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019087.926684:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019087.926685:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019087.926689:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3350 00000100:00080000:0.0:1734019087.926691:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019087.926695:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019087.926699:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019087.926703:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333093760/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019087.926706:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333094528/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019087.926713:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019087.926716:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019087.926721:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333093760:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019087.926728:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333093888:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019087.926745:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019087.926751:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019087/real 1734019087] req@ffff8800a705d180 x1818247333093888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019087.926762:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333093888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019087.926770:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019087.926772:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019087.926776:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333093888:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019087.926799:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333094016:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019087.926813:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019087.926819:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019087/real 1734019087] req@ffff8800a705e680 x1818247333094016/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019087.926829:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333094016/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019087.926837:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019087.926840:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019087.926843:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333094016:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019087.926849:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333094144:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019087.926861:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019087.926867:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019087/real 1734019087] req@ffff8800a705c700 x1818247333094144/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019087.926876:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333094144/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019087.926884:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019087.926886:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019087.926889:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333094144:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019087.926895:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333094272:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019087.926906:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019087.926912:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019087/real 1734019087] req@ffff8800a705f800 x1818247333094272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019087.926922:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333094272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019087.926930:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019087.926933:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019087.926935:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333094272:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019087.926940:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333094400:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019087.926955:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019087.926961:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019087/real 1734019087] req@ffff8800a705d500 x1818247333094400/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019087.926972:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333094400/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019087.926979:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019087.926981:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019087.926984:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333094400:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019087.926989:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333094528:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019087.927001:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019087.927007:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019087/real 1734019087] req@ffff8800a705d880 x1818247333094528/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019087.927017:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333094528/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019087.927024:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019087.927026:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019087.927029:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333094528:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019088.880532:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019089.882526:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019090.884512:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019091.886506:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019092.888489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019092.934332:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019092.934342:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019092.934346:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019092.934357:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3355 00000100:00080000:0.0:1734019092.934366:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019092.934375:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019092.934450:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333094656/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019092.934495:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019092.934501:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019092.934504:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019092.934510:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3355 00000100:00080000:0.0:1734019092.934516:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019092.934522:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019092.934534:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019092.934553:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333094784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019092.934568:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019092.934573:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1734019092.934574:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333094656:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019092.934576:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019092.934581:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3355 00000100:00080000:0.0:1734019092.934585:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019092.934590:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019092.934597:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019092.934609:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333094912/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019092.934621:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019092.934625:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019092.934627:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019092.934632:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3355 00000100:00080000:0.0:1734019092.934635:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019092.934640:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734019092.934643:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734019092.934647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019092.934659:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333095040/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019092.934667:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019092/real 1734019092] req@ffff8800a705e300 x1818247333094656/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019092.934672:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019092.934677:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019092.934680:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019092.934685:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3355 00000100:00080000:0.0:1734019092.934690:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019092.934694:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019092.934700:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1734019092.934701:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333094656/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019092.934712:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333095168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019092.934716:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019092.934721:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:0.0:1734019092.934725:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019092.934727:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333094656:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019092.934730:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019092.934733:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019092.934737:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3355 00000100:00080000:0.0:1734019092.934741:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019092.934746:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019092.934752:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019092.934752:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333094784:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019092.934765:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333095296/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019092.934778:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019092.934780:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019092.934787:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1734019092.934788:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019092/real 1734019092] req@ffff8800a705c000 x1818247333094784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019092.934789:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019092.934795:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3355 00000100:00080000:0.0:1734019092.934799:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019092.934804:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734019092.934806:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333094784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734019092.934812:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1734019092.934818:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019092.934822:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019092.934824:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333095424/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019092.934828:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333094784:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019092.934837:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333094912:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019092.934861:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019092.934873:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019092/real 1734019092] req@ffff8800a705dc00 x1818247333094912/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019092.934891:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333094912/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019092.934904:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019092.934908:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019092.934913:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333094912:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019092.934921:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333095040:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019092.934943:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019092.934950:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019092/real 1734019092] req@ffff8800a705f100 x1818247333095040/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019092.934961:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333095040/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019092.934969:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019092.934971:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019092.934974:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333095040:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019092.934979:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333095168:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019092.934991:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019092.934997:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019092/real 1734019092] req@ffff8800a705df80 x1818247333095168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019092.935007:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333095168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019092.935014:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019092.935016:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019092.935020:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333095168:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019092.935038:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333095296:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019092.935053:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019092.935059:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019092/real 1734019092] req@ffff8800a705ca80 x1818247333095296/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019092.935070:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333095296/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019092.935077:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019092.935079:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019092.935082:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333095296:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019092.935087:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333095424:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019092.935098:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019092.935104:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019092/real 1734019092] req@ffff8800a705ed80 x1818247333095424/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019092.935114:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333095424/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019092.935121:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019092.935123:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019092.935126:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333095424:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019093.890454:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019094.892524:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019095.894497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019096.896515:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019097.898520:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019097.942397:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019097.942407:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019097.942412:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019097.942423:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3360 00000100:00080000:0.0:1734019097.942438:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019097.942446:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019097.942483:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333095552/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019097.942518:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019097.942522:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019097.942524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019097.942529:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3360 00000100:00080000:0.0:1734019097.942532:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019097.942536:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019097.942545:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019097.942556:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333095680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019097.942567:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019097.942571:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019097.942572:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019097.942576:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3360 00000100:00080000:0.0:1734019097.942578:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019097.942582:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019097.942586:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019097.942596:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333095808/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019097.942605:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019097.942608:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019097.942609:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019097.942613:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3360 00000100:00080000:0.0:1734019097.942615:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019097.942619:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019097.942623:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019097.942631:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333095936/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019097.942639:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333095552:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019097.942640:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019097.942643:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019097.942644:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019097.942648:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3360 00000100:00080000:0.0:1734019097.942651:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019097.942655:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019097.942659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019097.942667:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333096064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019097.942676:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019097.942679:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019097.942680:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019097.942683:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3360 00000100:00080000:0.0:1734019097.942696:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019097.942699:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019097.942704:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019097.942711:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333096192/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019097.942719:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019097.942722:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019097.942724:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019097.942727:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3360 00000100:00080000:0.0:1734019097.942730:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019097.942733:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:1.0:1734019097.942737:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333095680:192.168.201.152@tcp:38:kworker.0 00000008:00100000:0.0:1734019097.942738:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019097.942746:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333096320/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019097.942758:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333095808:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019097.942781:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333095936:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019097.942796:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333096064:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019097.942810:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333096192:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019097.942825:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333096320:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019098.900510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019099.902512:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019100.904488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734019100.948794:0:1957:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734019100.948856:0:1957:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734019100.948878:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019100.948928:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019100.948941:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019100.948951:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019100.948962:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019100.948971:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019100.948982:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734019100.949005:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019097/real 1734019100] req@ffff8800a705ca80 x1818247333095936/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019100.949056:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333095936/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019100.949073:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019100.949079:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019100.949093:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333095936:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019100.949106:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019097/real 1734019100] req@ffff8800a705df80 x1818247333096064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019100.949126:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333096064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019100.949139:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019100.949143:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019100.949148:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333096064:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019100.949159:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019097/real 1734019100] req@ffff8800a705f100 x1818247333096192/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019100.949176:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333096192/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019100.949190:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019100.949195:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019100.949201:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333096192:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019100.949210:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019097/real 1734019100] req@ffff8800a705dc00 x1818247333096320/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019100.949337:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333096320/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019100.949350:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019100.949354:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019100.949360:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333096320:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019100.949370:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019097/real 1734019100] req@ffff8800a705fb80 x1818247333095552/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019100.949388:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333095552/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019100.949401:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019100.949405:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019100.949410:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333095552:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019100.949420:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019097/real 1734019100] req@ffff8800a705ea00 x1818247333095680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019100.949438:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333095680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019100.949461:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019100.949464:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019100.949470:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333095680:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019100.949479:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019097/real 1734019100] req@ffff8800a705ed80 x1818247333095808/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019100.949497:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333095808/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019100.949510:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019100.949514:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019100.949519:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333095808:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019101.906480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019102.908508:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019102.950340:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019102.950349:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019102.950353:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019102.950363:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3365 00000100:00080000:0.0:1734019102.950374:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019102.950381:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019102.950417:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333096448/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019102.950451:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019102.950455:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019102.950457:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019102.950462:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3365 00000100:00080000:0.0:1734019102.950465:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019102.950469:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019102.950478:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019102.950490:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333096576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019102.950500:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019102.950504:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019102.950506:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019102.950509:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3365 00000100:00080000:0.0:1734019102.950512:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019102.950515:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019102.950520:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019102.950528:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333096704/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019102.950536:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019102.950540:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019102.950541:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019102.950544:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3365 00000100:00080000:0.0:1734019102.950547:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019102.950550:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019102.950554:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019102.950563:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333096832/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019102.950567:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333096448:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019102.950571:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019102.950574:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019102.950576:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019102.950580:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3365 00000100:00080000:0.0:1734019102.950582:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019102.950585:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019102.950589:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019102.950597:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333096960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019102.950605:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019102.950608:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019102.950609:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019102.950613:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3365 00000100:00080000:0.0:1734019102.950625:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019102.950628:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019102.950633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019102.950635:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019102.950643:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333097088/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019102.950651:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019102.950654:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019102.950656:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019102.950656:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019102/real 1734019102] req@ffff8800a705c000 x1818247333096448/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019102.950659:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3365 00000100:00080000:0.0:1734019102.950662:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019102.950666:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019102.950670:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019102.950675:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333096448/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019102.950678:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333097216/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019102.950686:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019102.950689:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019102.950694:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333096448:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019102.950701:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333096576:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019102.950718:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019102.950725:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019102/real 1734019102] req@ffff8800a705e300 x1818247333096576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019102.950737:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333096576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019102.950744:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019102.950747:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019102.950750:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333096576:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019102.950756:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333096704:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019102.950769:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019102.950775:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019102/real 1734019102] req@ffff8800a705d880 x1818247333096704/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019102.950785:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333096704/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019102.950793:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019102.950795:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019102.950798:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333096704:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019102.950821:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333096832:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019102.950833:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019102.950839:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019102/real 1734019102] req@ffff8800a705d500 x1818247333096832/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019102.950850:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333096832/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019102.950865:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019102.950867:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019102.950870:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333096832:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019102.950875:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333096960:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019102.950887:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019102.950892:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019102/real 1734019102] req@ffff8800a705f800 x1818247333096960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019102.950902:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333096960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019102.950909:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019102.950912:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019102.950915:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333096960:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019102.950920:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333097088:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019102.950931:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019102.950937:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019102/real 1734019102] req@ffff8800a705c700 x1818247333097088/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019102.950947:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333097088/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019102.950954:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019102.950957:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019102.950960:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333097088:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019102.950966:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333097216:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019102.950978:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019102.950984:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019102/real 1734019102] req@ffff8800a705e680 x1818247333097216/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019102.950993:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333097216/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019102.951000:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019102.951002:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019102.951005:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333097216:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019103.910472:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019104.912487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019105.914495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019106.916503:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019107.918486:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019107.958345:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019107.958354:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019107.958358:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019107.958369:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3370 00000100:00080000:0.0:1734019107.958379:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019107.958386:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019107.958422:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333097344/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019107.958456:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019107.958461:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019107.958463:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019107.958467:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3370 00000100:00080000:0.0:1734019107.958470:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019107.958474:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019107.958484:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019107.958495:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333097472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019107.958505:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019107.958508:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019107.958510:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019107.958514:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3370 00000100:00080000:0.0:1734019107.958516:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019107.958520:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019107.958525:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019107.958536:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333097600/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019107.958545:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019107.958548:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019107.958549:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019107.958552:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3370 00000100:00080000:0.0:1734019107.958555:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019107.958558:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019107.958562:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019107.958570:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333097728/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019107.958570:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333097344:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019107.958579:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019107.958582:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019107.958584:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019107.958587:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3370 00000100:00080000:0.0:1734019107.958591:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019107.958594:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019107.958599:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019107.958606:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333097856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019107.958615:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019107.958618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019107.958620:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019107.958624:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3370 00000100:00080000:0.0:1734019107.958627:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019107.958631:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019107.958635:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019107.958638:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019107.958642:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333097984/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019107.958650:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019107.958653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019107.958654:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019107.958657:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3370 00000100:00080000:0.0:1734019107.958660:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019107.958661:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019107/real 1734019107] req@ffff8800a705d180 x1818247333097344/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019107.958663:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019107.958667:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019107.958675:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333098112/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019107.958681:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333097344/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019107.958691:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019107.958694:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019107.958699:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333097344:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019107.958706:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333097472:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019107.958722:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019107.958728:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019107/real 1734019107] req@ffff8800a705c380 x1818247333097472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019107.958749:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333097472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019107.958757:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019107.958759:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019107.958762:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333097472:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019107.958768:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333097600:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019107.958781:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019107.958787:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019107/real 1734019107] req@ffff8800a705e680 x1818247333097600/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019107.958797:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333097600/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019107.958804:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019107.958807:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019107.958809:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333097600:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019107.958834:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333097728:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019107.958846:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019107.958852:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019107/real 1734019107] req@ffff8800a705c700 x1818247333097728/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019107.958862:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333097728/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019107.958870:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019107.958873:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019107.958876:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333097728:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019107.958881:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333097856:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019107.958893:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019107.958899:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019107/real 1734019107] req@ffff8800a705f800 x1818247333097856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019107.958909:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333097856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019107.958917:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019107.958919:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019107.958922:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333097856:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019107.958927:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333097984:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019107.958938:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019107.958944:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019107/real 1734019107] req@ffff8800a705d500 x1818247333097984/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019107.958953:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333097984/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019107.958960:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019107.958962:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019107.958965:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333097984:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019107.958971:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333098112:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019107.958986:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019107.958992:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019107/real 1734019107] req@ffff8800a705d880 x1818247333098112/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019107.959002:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333098112/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019107.959009:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019107.959011:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019107.959014:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333098112:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019108.920471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019109.922500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019110.924473:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019111.926520:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019112.928494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019112.966348:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019112.966361:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019112.966367:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019112.966381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3375 00000100:00080000:0.0:1734019112.966393:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019112.966403:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019112.966449:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333098240/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019112.966494:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019112.966499:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019112.966501:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019112.966506:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3375 00000100:00080000:0.0:1734019112.966510:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019112.966514:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019112.966526:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019112.966538:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333098368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019112.966548:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019112.966552:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019112.966554:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019112.966558:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3375 00000100:00080000:0.0:1734019112.966561:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019112.966564:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019112.966570:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019112.966579:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333098496/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019112.966587:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019112.966590:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019112.966592:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019112.966595:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3375 00000100:00080000:0.0:1734019112.966598:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019112.966602:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019112.966607:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019112.966612:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333098240:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019112.966616:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333098624/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019112.966626:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019112.966629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019112.966631:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019112.966634:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3375 00000100:00080000:0.0:1734019112.966637:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019112.966640:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019112.966645:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019112.966653:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333098752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019112.966661:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019112.966665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019112.966666:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019112.966670:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3375 00000100:00080000:0.0:1734019112.966673:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019112.966677:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019112.966682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019112.966684:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019112.966693:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333098880/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019112.966702:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019112.966706:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019112.966708:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019112.966708:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019112/real 1734019112] req@ffff8800a705e300 x1818247333098240/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019112.966712:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3375 00000100:00080000:0.0:1734019112.966715:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019112.966719:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019112.966723:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019112.966728:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333098240/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019112.966731:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333099008/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019112.966740:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019112.966744:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019112.966749:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333098240:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019112.966757:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333098368:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019112.966776:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019112.966783:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019112/real 1734019112] req@ffff8800a705c000 x1818247333098368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019112.966794:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333098368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019112.966803:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019112.966806:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019112.966810:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333098368:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019112.966816:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333098496:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019112.966831:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019112.966837:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019112/real 1734019112] req@ffff8800a705ed80 x1818247333098496/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019112.966847:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333098496/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019112.966854:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019112.966857:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019112.966860:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333098496:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019112.966887:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333098624:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019112.966899:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019112.966905:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019112/real 1734019112] req@ffff8800a705ea00 x1818247333098624/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019112.966916:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333098624/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019112.966923:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019112.966926:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019112.966928:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333098624:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019112.966935:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333098752:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019112.966948:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019112.966954:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019112/real 1734019112] req@ffff8800a705fb80 x1818247333098752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019112.966964:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333098752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019112.966972:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019112.966974:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019112.966977:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333098752:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019112.966983:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333098880:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019112.966995:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019112.967000:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019112/real 1734019112] req@ffff8800a705dc00 x1818247333098880/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019112.967010:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333098880/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019112.967018:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019112.967020:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019112.967023:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333098880:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019112.967029:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333099008:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019112.967041:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019112.967047:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019112/real 1734019112] req@ffff8800a705f100 x1818247333099008/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019112.967058:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333099008/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019112.967101:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019112.967104:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019112.967108:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333099008:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019113.930497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019114.932465:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019115.934403:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019116.936498:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019117.938497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019117.974320:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019117.974329:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019117.974334:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019117.974344:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3380 00000100:00080000:0.0:1734019117.974354:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019117.974362:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019117.974399:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333099136/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019117.974433:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019117.974437:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019117.974439:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019117.974444:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3380 00000100:00080000:0.0:1734019117.974447:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019117.974451:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019117.974461:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019117.974472:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333099264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019117.974483:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019117.974486:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019117.974488:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019117.974491:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3380 00000100:00080000:0.0:1734019117.974494:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019117.974497:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019117.974502:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019117.974511:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333099392/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019117.974519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019117.974523:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019117.974524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019117.974527:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3380 00000100:00080000:0.0:1734019117.974530:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019117.974533:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019117.974538:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019117.974546:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333099520/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019117.974548:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333099136:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019117.974555:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019117.974559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019117.974560:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019117.974564:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3380 00000100:00080000:0.0:1734019117.974568:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019117.974571:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019117.974575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019117.974583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333099648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019117.974591:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019117.974594:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019117.974595:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019117.974599:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3380 00000100:00080000:0.0:1734019117.974602:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019117.974605:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019117.974609:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019117.974615:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019117.974617:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333099776/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019117.974624:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019117.974628:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019117.974629:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019117.974632:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3380 00000100:00080000:0.0:1734019117.974635:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019117.974636:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019117/real 1734019117] req@ffff8800a705df80 x1818247333099136/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019117.974638:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019117.974642:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019117.974650:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333099904/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019117.974655:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333099136/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019117.974666:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019117.974669:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019117.974674:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333099136:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019117.974681:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333099264:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019117.974698:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019117.974704:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019117/real 1734019117] req@ffff8800a705ca80 x1818247333099264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019117.974716:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333099264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019117.974724:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019117.974727:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019117.974731:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333099264:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019117.974737:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333099392:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019117.974751:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019117.974757:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019117/real 1734019117] req@ffff8800a705f100 x1818247333099392/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019117.974767:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333099392/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019117.974775:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019117.974777:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019117.974781:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333099392:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019117.974805:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333099520:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019117.974818:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019117.974823:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019117/real 1734019117] req@ffff8800a705dc00 x1818247333099520/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019117.974834:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333099520/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019117.974842:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019117.974844:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019117.974847:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333099520:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019117.974852:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333099648:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019117.974868:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019117.974874:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019117/real 1734019117] req@ffff8800a705fb80 x1818247333099648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019117.974884:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333099648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019117.974891:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019117.974893:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019117.974896:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333099648:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019117.974902:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333099776:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019117.974913:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019117.974919:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019117/real 1734019117] req@ffff8800a705ea00 x1818247333099776/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019117.974929:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333099776/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019117.974936:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019117.974938:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019117.974941:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333099776:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019117.974946:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333099904:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019117.974959:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019117.974965:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019117/real 1734019117] req@ffff8800a705ed80 x1818247333099904/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019117.975030:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333099904/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019117.975039:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019117.975041:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019117.975045:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333099904:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019118.940482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019119.942479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019120.944423:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019121.946473:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019122.948479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019122.982361:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019122.982371:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019122.982375:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019122.982385:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3385 00000100:00080000:0.0:1734019122.982395:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019122.982403:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019122.982437:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333100032/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019122.982470:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019122.982474:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019122.982476:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019122.982481:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3385 00000100:00080000:0.0:1734019122.982484:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019122.982489:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019122.982498:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019122.982509:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333100160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019122.982519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019122.982522:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019122.982524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019122.982528:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3385 00000100:00080000:0.0:1734019122.982530:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019122.982534:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019122.982540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019122.982548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333100288/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019122.982557:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019122.982560:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019122.982562:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019122.982565:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3385 00000100:00080000:0.0:1734019122.982567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019122.982571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019122.982575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019122.982577:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333100032:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019122.982583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333100416/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019122.982592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019122.982595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019122.982597:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019122.982600:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3385 00000100:00080000:0.0:1734019122.982602:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019122.982606:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019122.982610:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019122.982618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333100544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019122.982626:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019122.982629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019122.982631:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019122.982635:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3385 00000100:00080000:0.0:1734019122.982638:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019122.982641:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019122.982645:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019122.982647:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019122.982653:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333100672/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019122.982661:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019122.982664:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019122.982665:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019122.982668:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019122/real 1734019122] req@ffff8800a705c000 x1818247333100032/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019122.982669:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3385 00000100:00080000:0.0:1734019122.982671:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019122.982674:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019122.982678:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019122.982687:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333100800/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019122.982687:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333100032/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019122.982698:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019122.982701:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019122.982706:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333100032:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019122.982714:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333100160:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019122.982731:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019122.982738:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019122/real 1734019122] req@ffff8800a705e300 x1818247333100160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019122.982749:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333100160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019122.982756:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019122.982760:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019122.982763:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333100160:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019122.982770:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333100288:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019122.982783:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019122.982789:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019122/real 1734019122] req@ffff8800a705d880 x1818247333100288/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019122.982799:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333100288/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019122.982807:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019122.982810:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019122.982813:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333100288:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019122.982836:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333100416:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019122.982849:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019122.982854:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019122/real 1734019122] req@ffff8800a705d500 x1818247333100416/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019122.982865:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333100416/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019122.982873:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019122.982875:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019122.982878:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333100416:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019122.982883:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333100544:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019122.982951:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019122.982960:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019122/real 1734019122] req@ffff8800a705f800 x1818247333100544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019122.982971:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333100544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019122.982979:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019122.982981:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019122.982984:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333100544:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019122.982989:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333100672:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019122.983002:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019122.983008:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019122/real 1734019122] req@ffff8800a705c700 x1818247333100672/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019122.983018:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333100672/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019122.983026:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019122.983028:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019122.983031:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333100672:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019122.983036:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333100800:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019122.983047:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019122.983053:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019122/real 1734019122] req@ffff8800a705e680 x1818247333100800/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019122.983069:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333100800/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019122.983077:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019122.983079:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019122.983082:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333100800:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019123.950429:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019124.952449:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019125.954467:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019126.956468:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019127.958517:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019127.990349:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019127.990359:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019127.990363:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019127.990373:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3390 00000100:00080000:0.0:1734019127.990383:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019127.990391:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019127.990427:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333100928/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019127.990470:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019127.990475:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019127.990477:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019127.990481:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3390 00000100:00080000:0.0:1734019127.990485:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019127.990489:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019127.990499:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019127.990509:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333101056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019127.990519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019127.990522:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019127.990524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019127.990528:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3390 00000100:00080000:0.0:1734019127.990530:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019127.990534:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019127.990539:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019127.990548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333101184/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019127.990557:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019127.990560:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019127.990561:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019127.990565:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3390 00000100:00080000:0.0:1734019127.990567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019127.990571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019127.990576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019127.990579:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333100928:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019127.990583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333101312/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019127.990592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019127.990595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019127.990597:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019127.990600:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3390 00000100:00080000:0.0:1734019127.990603:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019127.990606:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019127.990611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019127.990618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333101440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019127.990626:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019127.990629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019127.990631:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019127.990634:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3390 00000100:00080000:0.0:1734019127.990637:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019127.990640:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019127.990644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019127.990645:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019127.990652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333101568/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019127.990660:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019127.990664:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019127.990666:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019127.990667:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019127/real 1734019127] req@ffff8800a705c380 x1818247333100928/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019127.990670:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3390 00000100:00080000:0.0:1734019127.990672:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019127.990676:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019127.990680:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019127.990686:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333100928/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019127.990687:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333101696/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019127.990697:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019127.990701:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019127.990705:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333100928:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019127.990713:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333101056:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019127.990730:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019127.990736:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019127/real 1734019127] req@ffff8800a705d180 x1818247333101056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019127.990747:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333101056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019127.990756:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019127.990759:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019127.990763:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333101056:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019127.990769:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333101184:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019127.990782:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019127.990788:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019127/real 1734019127] req@ffff8800a705e680 x1818247333101184/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019127.990798:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333101184/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019127.990806:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019127.990862:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019127.990867:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333101184:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019127.990891:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333101312:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019127.990909:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019127.990916:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019127/real 1734019127] req@ffff8800a705c700 x1818247333101312/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019127.990927:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333101312/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019127.990936:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019127.990938:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019127.990941:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333101312:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019127.990946:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333101440:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019127.990961:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019127.990967:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019127/real 1734019127] req@ffff8800a705f800 x1818247333101440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019127.990978:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333101440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019127.990985:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019127.990987:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019127.990990:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333101440:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019127.990995:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333101568:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019127.991007:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019127.991013:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019127/real 1734019127] req@ffff8800a705d500 x1818247333101568/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019127.991022:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333101568/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019127.991029:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019127.991031:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019127.991034:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333101568:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019127.991040:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333101696:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019127.991051:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019127.991057:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019127/real 1734019127] req@ffff8800a705d880 x1818247333101696/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019127.991073:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333101696/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019127.991080:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019127.991083:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019127.991086:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333101696:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019128.960450:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019129.962351:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019130.964499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019131.966478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019132.968471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019132.998338:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019132.998348:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019132.998352:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019132.998362:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3395 00000100:00080000:0.0:1734019132.998373:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019132.998380:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019132.998414:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333101824/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019132.998447:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019132.998451:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019132.998454:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019132.998458:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3395 00000100:00080000:0.0:1734019132.998461:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019132.998465:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019132.998475:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019132.998486:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333101952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019132.998497:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019132.998501:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019132.998502:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019132.998506:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3395 00000100:00080000:0.0:1734019132.998508:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019132.998512:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019132.998516:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019132.998524:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333102080/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019132.998533:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019132.998536:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019132.998538:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019132.998541:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3395 00000100:00080000:0.0:1734019132.998543:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019132.998547:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019132.998551:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019132.998557:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333101824:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019132.998559:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333102208/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019132.998568:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019132.998571:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019132.998572:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019132.998576:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3395 00000100:00080000:0.0:1734019132.998579:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019132.998582:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019132.998586:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019132.998594:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333102336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019132.998602:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019132.998605:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019132.998607:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019132.998610:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3395 00000100:00080000:0.0:1734019132.998613:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019132.998616:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019132.998620:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019132.998624:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019132.998629:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333102464/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019132.998637:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019132.998640:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019132.998642:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019132.998645:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3395 00000100:00100000:1.0:1734019132.998645:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019132/real 1734019132] req@ffff8800a705e300 x1818247333101824/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019132.998648:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019132.998651:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019132.998655:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019132.998662:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333102592/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019132.998664:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333101824/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019132.998676:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019132.998679:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019132.998684:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333101824:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019132.998691:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333101952:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019132.998708:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019132.998715:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019132/real 1734019132] req@ffff8800a705c000 x1818247333101952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019132.998768:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333101952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019132.998777:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019132.998781:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019132.998784:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333101952:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019132.998791:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333102080:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019132.998808:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019132.998814:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019132/real 1734019132] req@ffff8800a705ed80 x1818247333102080/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019132.998825:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333102080/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019132.998833:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019132.998835:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019132.998839:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333102080:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019132.998863:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333102208:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019132.998875:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019132.998881:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019132/real 1734019132] req@ffff8800a705ea00 x1818247333102208/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019132.998891:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333102208/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019132.998899:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019132.998901:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019132.998904:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333102208:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019132.998909:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333102336:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019132.998925:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019132.998931:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019132/real 1734019132] req@ffff8800a705fb80 x1818247333102336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019132.998941:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333102336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019132.998948:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019132.998950:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019132.998953:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333102336:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019132.998959:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333102464:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019132.998970:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019132.998976:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019132/real 1734019132] req@ffff8800a705dc00 x1818247333102464/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019132.998986:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333102464/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019132.998994:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019132.998996:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019132.998999:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333102464:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019132.999004:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333102592:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019132.999015:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019132.999021:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019132/real 1734019132] req@ffff8800a705f100 x1818247333102592/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019132.999037:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333102592/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019132.999045:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019132.999047:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019132.999050:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333102592:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019133.970496:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019134.972452:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019135.974468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019136.976495:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019137.978477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019138.006331:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019138.006340:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019138.006345:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019138.006355:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3400 00000100:00080000:0.0:1734019138.006370:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019138.006377:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019138.006413:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333102720/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019138.006448:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019138.006452:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019138.006454:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019138.006459:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3400 00000100:00080000:0.0:1734019138.006462:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019138.006466:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019138.006476:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019138.006487:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333102848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019138.006496:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019138.006500:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019138.006502:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019138.006506:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3400 00000100:00080000:0.0:1734019138.006508:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019138.006512:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019138.006518:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019138.006528:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333102976/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019138.006537:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019138.006540:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019138.006542:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019138.006545:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3400 00000100:00080000:0.0:1734019138.006547:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019138.006550:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019138.006555:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019138.006562:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333103104/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019138.006562:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333102720:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019138.006572:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019138.006575:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019138.006577:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019138.006580:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3400 00000100:00080000:0.0:1734019138.006583:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019138.006586:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019138.006590:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019138.006598:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333103232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019138.006606:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019138.006609:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019138.006611:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019138.006614:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3400 00000100:00080000:0.0:1734019138.006617:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019138.006620:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019138.006625:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019138.006628:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019138.006632:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333103360/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019138.006673:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019138.006677:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019138.006678:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019138.006682:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3400 00000100:00080000:0.0:1734019138.006685:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019138.006687:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019138/real 1734019138] req@ffff8800a705ca80 x1818247333102720/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019138.006688:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019138.006693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019138.006702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333103488/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019138.006707:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333102720/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019138.006717:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019138.006720:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019138.006725:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333102720:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019138.006732:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333102848:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019138.006751:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019138.006758:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019138/real 1734019138] req@ffff8800a705df80 x1818247333102848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019138.006769:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333102848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019138.006777:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019138.006781:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019138.006785:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333102848:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019138.006791:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333102976:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019138.006804:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019138.006810:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019138/real 1734019138] req@ffff8800a705f100 x1818247333102976/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019138.006820:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333102976/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019138.006827:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019138.006830:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019138.006834:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333102976:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019138.006858:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333103104:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019138.006870:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019138.006877:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019138/real 1734019138] req@ffff8800a705dc00 x1818247333103104/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019138.006888:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333103104/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019138.006896:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019138.006898:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019138.006901:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333103104:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019138.006905:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333103232:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019138.006920:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019138.006926:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019138/real 1734019138] req@ffff8800a705fb80 x1818247333103232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019138.006937:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333103232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019138.006944:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019138.006946:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019138.006949:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333103232:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019138.006953:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333103360:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019138.006965:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019138.006971:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019138/real 1734019138] req@ffff8800a705ea00 x1818247333103360/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019138.006980:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333103360/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019138.006987:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019138.006989:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019138.006992:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333103360:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019138.006997:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333103488:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019138.007009:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019138.007015:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019138/real 1734019138] req@ffff8800a705ed80 x1818247333103488/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019138.007031:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333103488/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019138.007039:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019138.007041:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019138.007044:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333103488:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019138.980463:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019139.982481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019140.984417:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019141.986458:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019142.988479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019143.014303:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019143.014313:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019143.014317:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019143.014328:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3405 00000100:00080000:0.0:1734019143.014338:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019143.014346:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019143.014381:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333103616/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019143.014423:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019143.014427:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019143.014429:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019143.014434:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3405 00000100:00080000:0.0:1734019143.014437:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019143.014441:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019143.014451:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019143.014463:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333103744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019143.014472:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019143.014475:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019143.014477:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019143.014481:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3405 00000100:00080000:0.0:1734019143.014483:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019143.014487:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019143.014492:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019143.014500:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333103872/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019143.014508:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019143.014511:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019143.014512:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019143.014516:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3405 00000100:00080000:0.0:1734019143.014518:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019143.014522:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019143.014526:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019143.014533:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333103616:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019143.014535:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333104000/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019143.014544:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019143.014547:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019143.014549:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019143.014583:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3405 00000100:00080000:0.0:1734019143.014588:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019143.014591:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019143.014596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019143.014605:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333104128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019143.014614:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019143.014618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019143.014619:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019143.014623:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3405 00000100:00080000:0.0:1734019143.014626:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019143.014629:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019143.014633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019143.014635:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019143.014641:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333104256/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019143.014649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019143.014653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019143.014655:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019143.014657:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019143/real 1734019143] req@ffff8800a705c000 x1818247333103616/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019143.014658:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3405 00000100:00080000:0.0:1734019143.014661:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019143.014665:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019143.014669:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019143.014677:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333103616/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019143.014678:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333104384/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019143.014687:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019143.014691:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019143.014695:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333103616:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019143.014703:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333103744:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019143.014721:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019143.014727:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019143/real 1734019143] req@ffff8800a705e300 x1818247333103744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019143.014738:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333103744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019143.014746:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019143.014749:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019143.014753:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333103744:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019143.014759:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333103872:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019143.014772:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019143.014778:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019143/real 1734019143] req@ffff8800a705d880 x1818247333103872/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019143.014788:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333103872/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019143.014796:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019143.014798:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019143.014801:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333103872:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019143.014826:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333104000:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019143.014838:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019143.014844:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019143/real 1734019143] req@ffff8800a705d500 x1818247333104000/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019143.014854:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333104000/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019143.014862:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019143.014865:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019143.014868:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333104000:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019143.014873:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333104128:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019143.014888:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019143.014894:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019143/real 1734019143] req@ffff8800a705f800 x1818247333104128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019143.014905:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333104128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019143.014912:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019143.014915:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019143.014918:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333104128:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019143.014923:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333104256:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019143.014934:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019143.014940:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019143/real 1734019143] req@ffff8800a705c700 x1818247333104256/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019143.014950:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333104256/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019143.014957:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019143.014959:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019143.014962:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333104256:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019143.014967:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333104384:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019143.014979:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019143.014985:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019143/real 1734019143] req@ffff8800a705e680 x1818247333104384/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019143.015002:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333104384/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019143.015009:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019143.015011:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019143.015014:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333104384:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019143.990503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019144.992515:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019145.994433:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019146.996560:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019147.998531:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019148.022323:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019148.022332:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019148.022337:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019148.022348:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3410 00000100:00080000:0.0:1734019148.022358:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019148.022366:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019148.022402:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333104512/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019148.022436:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019148.022440:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019148.022442:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019148.022447:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3410 00000100:00080000:0.0:1734019148.022450:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019148.022454:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019148.022464:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019148.022517:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333104640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019148.022529:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019148.022532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019148.022534:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019148.022537:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3410 00000100:00080000:0.0:1734019148.022540:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019148.022544:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019148.022549:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019148.022559:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333104768/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019148.022569:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019148.022570:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333104512:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019148.022572:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019148.022574:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019148.022577:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3410 00000100:00080000:0.0:1734019148.022580:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019148.022583:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019148.022587:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019148.022595:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333104896/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019148.022603:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019148.022606:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019148.022607:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019148.022611:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3410 00000100:00080000:0.0:1734019148.022624:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019148.022628:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019148.022633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1734019148.022637:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019148.022641:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333105024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019148.022650:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019148.022653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019148.022655:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019148.022659:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3410 00000100:00100000:1.0:1734019148.022659:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019148/real 1734019148] req@ffff8800a705d180 x1818247333104512/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019148.022661:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019148.022665:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019148.022669:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019148.022676:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333105152/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019148.022678:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333104512/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019148.022684:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019148.022687:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019148.022689:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1734019148.022689:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019148.022692:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3410 00000100:00080000:1.0:1734019148.022692:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:0.0:1734019148.022694:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1734019148.022697:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333104512:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019148.022698:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019148.022702:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019148.022705:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333104640:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019148.022711:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333105280/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:1.0:1734019148.022721:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019148.022728:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019148/real 1734019148] req@ffff8800a705c380 x1818247333104640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019148.022739:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333104640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019148.022748:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019148.022751:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019148.022755:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333104640:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019148.022778:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333104768:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019148.022792:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019148.022798:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019148/real 1734019148] req@ffff8800a705e680 x1818247333104768/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019148.022809:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333104768/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019148.022816:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019148.022818:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019148.022821:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333104768:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019148.022827:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333104896:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019148.022838:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019148.022844:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019148/real 1734019148] req@ffff8800a705c700 x1818247333104896/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019148.022854:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333104896/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019148.022861:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019148.022863:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019148.022866:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333104896:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019148.022871:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333105024:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019148.022887:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019148.022894:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019148/real 1734019148] req@ffff8800a705f800 x1818247333105024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019148.022904:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333105024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019148.022912:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019148.022914:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019148.022916:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333105024:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019148.022922:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333105152:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019148.022934:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019148.022940:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019148/real 1734019148] req@ffff8800a705d500 x1818247333105152/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019148.022949:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333105152/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019148.022956:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019148.022958:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019148.022961:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333105152:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019148.022967:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333105280:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019148.022978:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019148.022984:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019148/real 1734019148] req@ffff8800a705d880 x1818247333105280/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019148.023001:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333105280/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019148.023009:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019148.023011:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019148.023014:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333105280:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019149.000533:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019150.002429:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019151.004501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019152.006490:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019153.008488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019153.030324:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019153.030335:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019153.030340:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019153.030353:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3415 00000100:00080000:0.0:1734019153.030364:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019153.030373:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019153.030447:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333105408/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019153.030484:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019153.030488:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019153.030490:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019153.030495:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3415 00000100:00080000:0.0:1734019153.030499:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019153.030504:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019153.030516:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019153.030527:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333105536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019153.030538:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019153.030542:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019153.030543:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019153.030547:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3415 00000100:00080000:0.0:1734019153.030550:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019153.030554:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019153.030560:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019153.030570:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333105664/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019153.030579:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019153.030583:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019153.030584:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019153.030588:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3415 00000100:00080000:0.0:1734019153.030590:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019153.030594:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019153.030598:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019153.030606:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333105792/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019153.030616:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019153.030620:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1734019153.030620:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333105408:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019153.030621:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019153.030625:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3415 00000100:00080000:0.0:1734019153.030628:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019153.030632:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019153.030636:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019153.030646:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333105920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019153.030654:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019153.030658:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019153.030660:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019153.030665:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3415 00000100:00080000:0.0:1734019153.030684:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019153.030688:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019153.030693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019153.030697:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019153.030702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333106048/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019153.030711:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019153.030715:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019153.030716:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019153.030721:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3415 00000100:00100000:1.0:1734019153.030721:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019153/real 1734019153] req@ffff8800a705e300 x1818247333105408/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019153.030724:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019153.030728:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019153.030732:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019153.030740:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333106176/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019153.030742:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333105408/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019153.030755:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019153.030758:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019153.030763:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333105408:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019153.030772:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333105536:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019153.030790:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019153.030797:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019153/real 1734019153] req@ffff8800a705c000 x1818247333105536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019153.030809:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333105536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019153.030816:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019153.030820:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019153.030824:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333105536:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019153.030831:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333105664:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019153.030845:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019153.030851:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019153/real 1734019153] req@ffff8800a705ed80 x1818247333105664/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019153.030861:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333105664/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019153.030869:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019153.030872:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019153.030875:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333105664:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019153.030902:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333105792:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019153.030915:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019153.030921:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019153/real 1734019153] req@ffff8800a705ea00 x1818247333105792/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019153.030932:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333105792/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019153.030940:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019153.030942:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019153.030945:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333105792:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019153.030951:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333105920:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019153.030967:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019153.030973:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019153/real 1734019153] req@ffff8800a705fb80 x1818247333105920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019153.030983:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333105920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019153.030991:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019153.030994:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019153.030997:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333105920:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019153.031003:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333106048:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019153.031015:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019153.031021:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019153/real 1734019153] req@ffff8800a705dc00 x1818247333106048/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019153.031031:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333106048/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019153.031038:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019153.031040:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019153.031043:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333106048:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019153.031049:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333106176:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019153.031061:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019153.031067:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019153/real 1734019153] req@ffff8800a705f100 x1818247333106176/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019153.031084:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333106176/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019153.031092:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019153.031094:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019153.031097:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333106176:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019154.010465:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019155.012540:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019156.014446:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019157.016516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019158.018485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019158.038366:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019158.038372:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019158.038375:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019158.038382:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3420 00000100:00080000:0.0:1734019158.038390:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019158.038395:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019158.038441:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333106304/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019158.038484:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019158.038488:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019158.038490:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019158.038495:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3420 00000100:00080000:0.0:1734019158.038498:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019158.038502:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019158.038512:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019158.038523:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333106432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019158.038533:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019158.038537:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019158.038538:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019158.038542:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3420 00000100:00080000:0.0:1734019158.038544:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019158.038547:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019158.038552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019158.038562:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333106560/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019158.038570:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019158.038573:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019158.038575:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019158.038578:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3420 00000100:00080000:0.0:1734019158.038581:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019158.038585:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019158.038590:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019158.038597:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333106688/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019158.038606:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1734019158.038606:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333106304:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019158.038609:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019158.038611:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019158.038615:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3420 00000100:00080000:0.0:1734019158.038617:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019158.038620:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019158.038624:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019158.038632:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333106816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019158.038640:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019158.038643:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019158.038644:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019158.038648:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3420 00000100:00080000:0.0:1734019158.038659:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019158.038663:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019158.038667:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1734019158.038673:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019158.038676:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333106944/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019158.038684:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019158.038688:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019158.038690:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019158.038693:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3420 00000100:00100000:1.0:1734019158.038695:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019158/real 1734019158] req@ffff8800a705df80 x1818247333106304/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019158.038697:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019158.038700:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019158.038704:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019158.038712:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333107072/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019158.038714:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333106304/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019158.038724:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019158.038728:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019158.038732:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333106304:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019158.038739:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333106432:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019158.038757:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019158.038763:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019158/real 1734019158] req@ffff8800a705ca80 x1818247333106432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019158.038774:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333106432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019158.038782:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019158.038786:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019158.038789:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333106432:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019158.038795:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333106560:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019158.038809:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019158.038815:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019158/real 1734019158] req@ffff8800a705f100 x1818247333106560/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019158.038825:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333106560/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019158.038832:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019158.038835:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019158.038838:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333106560:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019158.038862:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333106688:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019158.038874:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019158.038880:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019158/real 1734019158] req@ffff8800a705dc00 x1818247333106688/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019158.038890:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333106688/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019158.038898:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019158.038900:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019158.038903:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333106688:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019158.038908:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333106816:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019158.038923:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019158.038929:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019158/real 1734019158] req@ffff8800a705fb80 x1818247333106816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019158.038939:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333106816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019158.038946:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019158.038948:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019158.038951:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333106816:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019158.038956:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333106944:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019158.038968:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019158.038974:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019158/real 1734019158] req@ffff8800a705ea00 x1818247333106944/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019158.038984:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333106944/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019158.038991:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019158.038993:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019158.038996:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333106944:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019158.039001:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333107072:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019158.039013:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019158.039018:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019158/real 1734019158] req@ffff8800a705ed80 x1818247333107072/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019158.039035:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333107072/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019158.039043:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019158.039045:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019158.039048:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333107072:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019159.020511:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019160.022482:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019161.024521:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019162.026507:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019163.028516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019163.046361:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019163.046374:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019163.046379:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019163.046394:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3425 00000100:00080000:0.0:1734019163.046407:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019163.046416:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019163.046466:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333107200/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019163.046505:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019163.046510:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019163.046512:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019163.046517:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3425 00000100:00080000:0.0:1734019163.046520:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019163.046525:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019163.046536:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019163.046548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333107328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019163.046561:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019163.046564:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019163.046566:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019163.046570:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3425 00000100:00080000:0.0:1734019163.046573:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019163.046576:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019163.046581:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019163.046590:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333107456/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019163.046599:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019163.046602:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019163.046604:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019163.046607:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3425 00000100:00080000:0.0:1734019163.046610:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019163.046613:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019163.046617:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019163.046627:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333107584/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019163.046631:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333107200:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019163.046637:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019163.046641:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019163.046643:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019163.046647:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3425 00000100:00080000:0.0:1734019163.046650:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019163.046654:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019163.046659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019163.046668:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333107712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019163.046676:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019163.046680:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019163.046681:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019163.046685:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3425 00000100:00080000:0.0:1734019163.046691:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019163.046695:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019163.046700:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019163.046707:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333107840/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019163.046715:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019163.046719:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019163.046720:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019163.046724:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3425 00000100:00080000:0.0:1734019163.046727:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019163.046730:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019163.046735:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019163.046736:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333107328:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019163.046746:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333107968/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:1.0:1734019163.046759:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333107456:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019163.046783:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333107584:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019163.046798:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333107712:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019163.046814:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333107840:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019163.046830:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333107968:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019164.030498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019165.032484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019166.034484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734019166.052694:0:1958:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734019166.052763:0:1958:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734019166.052787:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019166.052846:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019166.052858:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019166.052869:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019166.052880:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019166.052890:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019166.052900:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:1.0:1734019166.053072:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019163/real 1734019166] req@ffff8800a705d500 x1818247333107584/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019166.053112:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333107584/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019166.053130:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019166.053136:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019166.053143:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333107584:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019166.053155:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019163/real 1734019166] req@ffff8800a705f800 x1818247333107712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019166.053265:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333107712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019166.053281:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019166.053285:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019166.053292:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333107712:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019166.053303:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019163/real 1734019166] req@ffff8800a705c700 x1818247333107840/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019166.053331:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333107840/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019166.053343:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019166.053348:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019166.053353:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333107840:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019166.053363:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019163/real 1734019166] req@ffff8800a705e680 x1818247333107968/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019166.053380:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333107968/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019166.053392:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019166.053396:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019166.053401:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333107968:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019166.053415:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019163/real 1734019166] req@ffff8800a705c000 x1818247333107200/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019166.053434:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333107200/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019166.053446:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019166.053450:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019166.053455:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333107200:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019166.053465:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019163/real 1734019166] req@ffff8800a705e300 x1818247333107328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019166.053483:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333107328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019166.053496:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019166.053500:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019166.053505:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333107328:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019166.053515:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019163/real 1734019166] req@ffff8800a705d880 x1818247333107456/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019166.053534:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333107456/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019166.053547:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019166.053560:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019166.053566:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333107456:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019167.036437:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019168.038448:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019168.054343:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019168.054352:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019168.054357:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019168.054368:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3430 00000100:00080000:0.0:1734019168.054378:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019168.054386:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019168.054422:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333108096/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019168.054455:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019168.054459:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019168.054461:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019168.054466:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3430 00000100:00080000:0.0:1734019168.054469:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019168.054473:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019168.054483:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019168.054494:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333108224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019168.054504:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019168.054507:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019168.054509:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019168.054512:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3430 00000100:00080000:0.0:1734019168.054515:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019168.054518:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019168.054523:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019168.054533:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333108352/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019168.054542:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019168.054545:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019168.054546:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019168.054549:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3430 00000100:00080000:0.0:1734019168.054552:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019168.054556:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019168.054560:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019168.054562:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333108096:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019168.054569:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333108480/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019168.054577:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019168.054581:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019168.054582:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019168.054586:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3430 00000100:00080000:0.0:1734019168.054589:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019168.054592:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019168.054596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019168.054604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333108608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019168.054612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019168.054615:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019168.054616:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019168.054620:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3430 00000100:00080000:0.0:1734019168.054623:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019168.054627:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:1.0:1734019168.054630:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734019168.054631:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019168.054638:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333108736/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019168.054646:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019168.054650:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019168.054651:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1734019168.054651:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019168/real 1734019168] req@ffff8800a705c380 x1818247333108096/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019168.054654:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3430 00000100:00080000:0.0:1734019168.054657:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019168.054660:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019168.054665:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1734019168.054671:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333108096/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019168.054672:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333108864/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:1.0:1734019168.054683:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019168.054686:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019168.054691:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333108096:192.168.201.152@tcp:250:kworker.0 00000100:00100000:1.0:1734019168.054698:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333108224:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019168.054715:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019168.054722:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019168/real 1734019168] req@ffff8800a705d180 x1818247333108224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019168.054733:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333108224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019168.054741:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019168.054743:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019168.054747:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333108224:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019168.054752:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333108352:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019168.054765:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019168.054771:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019168/real 1734019168] req@ffff8800a705d880 x1818247333108352/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019168.054781:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333108352/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019168.054790:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019168.054793:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019168.054796:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333108352:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019168.054820:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333108480:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019168.054832:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019168.054838:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019168/real 1734019168] req@ffff8800a705e300 x1818247333108480/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019168.054848:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333108480/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019168.054856:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019168.054858:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019168.054861:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333108480:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019168.054866:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333108608:192.168.201.152@tcp:38:kworker.0 00000800:00000100:1.0:1734019168.054877:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019168.054883:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019168/real 1734019168] req@ffff8800a705c000 x1818247333108608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019168.054892:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333108608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019168.054900:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019168.054902:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019168.054905:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333108608:192.168.201.152@tcp:38:kworker.0 00000100:00100000:1.0:1734019168.054911:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333108736:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019168.054923:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019168.054929:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019168/real 1734019168] req@ffff8800a705e680 x1818247333108736/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019168.054938:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333108736/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019168.054945:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019168.054947:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019168.054950:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333108736:192.168.201.152@tcp:8:kworker.0 00000100:00100000:1.0:1734019168.054955:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333108864:192.168.201.152@tcp:8:kworker.0 00000800:00000100:1.0:1734019168.054967:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:1.0:1734019168.054973:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019168/real 1734019168] req@ffff8800a705c700 x1818247333108864/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:1.0:1734019168.054982:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333108864/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:1.0:1734019168.054989:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1734019168.054992:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:1.0:1734019168.054995:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333108864:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019169.040508:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019170.042453:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019171.044484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019172.046483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019173.048499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019173.062345:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019173.062356:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019173.062362:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019173.062377:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3435 00000100:00080000:0.0:1734019173.062389:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019173.062399:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019173.062448:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333108992/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019173.062488:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019173.062492:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019173.062495:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019173.062500:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3435 00000100:00080000:0.0:1734019173.062503:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019173.062507:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019173.062519:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019173.062531:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333109120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019173.062533:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333108992:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019173.062542:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019173.062545:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019173.062547:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019173.062552:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3435 00000100:00080000:0.0:1734019173.062555:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019173.062558:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019173.062564:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019173.062574:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333109248/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019173.062583:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019173.062586:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019173.062588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019173.062592:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3435 00000100:00080000:0.0:1734019173.062594:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019173.062598:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019173.062603:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019173.062611:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333109376/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019173.062619:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019173.062622:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1734019173.062622:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019173.062624:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019173.062627:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3435 00000100:00080000:0.0:1734019173.062631:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019173.062634:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019173.062639:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019173.062648:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333109504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019173.062648:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019173/real 1734019173] req@ffff8800a705f800 x1818247333108992/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019173.062657:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019173.062661:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019173.062663:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019173.062669:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3435 00000100:00100000:2.0:1734019173.062669:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333108992/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019173.062672:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019173.062675:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019173.062680:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1734019173.062681:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019173.062685:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019173.062688:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333109632/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019173.062690:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333108992:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019173.062696:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019173.062699:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019173.062700:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019173.062704:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3435 00000100:00080000:0.0:1734019173.062707:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019173.062710:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019173.062714:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019173.062717:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333109120:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019173.062723:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333109760/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734019173.062736:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019173.062743:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019173/real 1734019173] req@ffff8800a705d500 x1818247333109120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019173.062755:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333109120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019173.062763:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019173.062765:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019173.062769:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333109120:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019173.062774:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333109248:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019173.062789:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019173.062795:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019173/real 1734019173] req@ffff8800a705ed80 x1818247333109248/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019173.062806:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333109248/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019173.062814:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019173.062817:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019173.062820:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333109248:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019173.062825:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333109376:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019173.062838:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019173.062843:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019173/real 1734019173] req@ffff8800a705ea00 x1818247333109376/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019173.062853:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333109376/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019173.062861:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019173.062863:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019173.062866:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333109376:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019173.062871:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333109504:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019173.062884:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019173.062890:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019173/real 1734019173] req@ffff8800a705fb80 x1818247333109504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019173.062900:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333109504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019173.062907:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019173.062910:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019173.062913:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333109504:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019173.062919:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333109632:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019173.062932:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019173.062937:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019173/real 1734019173] req@ffff8800a705dc00 x1818247333109632/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019173.062947:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333109632/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019173.062955:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019173.062957:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019173.062961:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333109632:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019173.062978:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333109760:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019173.062990:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019173.062996:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019173/real 1734019173] req@ffff8800a705f100 x1818247333109760/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019173.063006:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333109760/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019173.063015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019173.063017:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019173.063020:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333109760:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019174.050502:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019175.052498:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019176.054509:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019177.056513:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019178.058506:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019178.070347:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019178.070356:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019178.070361:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019178.070373:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3440 00000100:00080000:0.0:1734019178.070383:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019178.070390:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019178.070427:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333109888/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019178.070469:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019178.070473:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019178.070475:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019178.070479:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3440 00000100:00080000:0.0:1734019178.070483:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019178.070486:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019178.070496:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019178.070507:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333110016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019178.070517:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019178.070520:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019178.070522:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019178.070526:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3440 00000100:00080000:0.0:1734019178.070528:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019178.070531:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019178.070536:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019178.070546:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333110144/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019178.070554:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019178.070557:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019178.070559:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019178.070562:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3440 00000100:00080000:0.0:1734019178.070565:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019178.070569:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019178.070573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019178.070582:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333110272/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019178.070583:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333109888:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019178.070591:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019178.070594:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019178.070596:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019178.070599:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3440 00000100:00080000:0.0:1734019178.070601:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019178.070605:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019178.070609:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019178.070616:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333110400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019178.070624:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019178.070627:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019178.070629:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019178.070632:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3440 00000100:00080000:0.0:1734019178.070635:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019178.070638:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019178.070642:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019178.070650:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333110528/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734019178.070656:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019178.070658:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019178.070662:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019178.070663:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019178.070667:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3440 00000100:00080000:0.0:1734019178.070670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019178.070673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019178.070677:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019178.070679:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019178/real 1734019178] req@ffff8800a705ca80 x1818247333109888/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019178.070684:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333110656/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019178.070698:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333109888/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019178.070710:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019178.070713:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019178.070717:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333109888:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019178.070724:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333110016:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019178.070741:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019178.070747:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019178/real 1734019178] req@ffff8800a705df80 x1818247333110016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019178.070758:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333110016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019178.070767:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019178.070769:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019178.070773:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333110016:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019178.070778:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333110144:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019178.070792:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019178.070797:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019178/real 1734019178] req@ffff8800a705f100 x1818247333110144/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019178.070807:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333110144/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019178.070816:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019178.070819:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019178.070822:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333110144:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019178.070846:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333110272:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019178.070858:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019178.070864:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019178/real 1734019178] req@ffff8800a705dc00 x1818247333110272/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019178.070875:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333110272/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019178.070883:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019178.070885:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019178.070888:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333110272:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019178.070893:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333110400:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019178.070904:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019178.070909:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019178/real 1734019178] req@ffff8800a705fb80 x1818247333110400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019178.070919:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333110400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019178.070927:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019178.070929:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019178.070932:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333110400:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019178.070937:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333110528:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019178.070949:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019178.070954:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019178/real 1734019178] req@ffff8800a705ea00 x1818247333110528/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019178.071016:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333110528/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019178.071024:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019178.071026:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019178.071029:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333110528:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019178.071035:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333110656:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019178.071052:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019178.071059:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019178/real 1734019178] req@ffff8800a705ed80 x1818247333110656/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019178.071069:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333110656/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019178.071077:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019178.071079:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019178.071082:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333110656:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019179.060491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019180.062478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019181.064481:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019182.066506:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019183.068507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019183.078364:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019183.078376:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019183.078381:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019183.078395:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3445 00000100:00080000:0.0:1734019183.078408:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019183.078418:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019183.078465:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333110784/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019183.078503:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019183.078508:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019183.078510:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019183.078515:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3445 00000100:00080000:0.0:1734019183.078519:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019183.078523:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019183.078534:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019183.078546:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333110912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019183.078558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019183.078561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019183.078563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019183.078567:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3445 00000100:00080000:0.0:1734019183.078570:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019183.078573:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019183.078578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019183.078587:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333111040/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019183.078595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019183.078598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019183.078600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019183.078603:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3445 00000100:00080000:0.0:1734019183.078606:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019183.078609:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019183.078614:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019183.078622:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333111168/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019183.078630:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333110784:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019183.078632:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019183.078636:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019183.078637:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019183.078642:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3445 00000100:00080000:0.0:1734019183.078645:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019183.078649:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019183.078654:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019183.078663:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333111296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019183.078671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019183.078674:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019183.078675:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019183.078679:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3445 00000100:00080000:0.0:1734019183.078682:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019183.078685:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019183.078690:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019183.078698:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333111424/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019183.078706:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1734019183.078707:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019183.078709:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019183.078711:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019183.078714:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3445 00000100:00080000:0.0:1734019183.078717:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019183.078721:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019183.078725:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019183.078732:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019183/real 1734019183] req@ffff8800a705d500 x1818247333110784/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019183.078734:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333111552/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019183.078754:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333110784/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019183.078767:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019183.078770:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019183.078775:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333110784:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019183.078784:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333110912:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019183.078802:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019183.078809:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019183/real 1734019183] req@ffff8800a705f800 x1818247333110912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019183.078821:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333110912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019183.078829:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019183.078833:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019183.078836:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333110912:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019183.078842:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333111040:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019183.078856:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019183.078862:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019183/real 1734019183] req@ffff8800a705c700 x1818247333111040/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019183.078919:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333111040/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019183.078928:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019183.078931:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019183.078935:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333111040:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019183.078963:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333111168:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019183.078981:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019183.078988:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019183/real 1734019183] req@ffff8800a705e680 x1818247333111168/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019183.078999:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333111168/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019183.079007:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019183.079009:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019183.079012:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333111168:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019183.079018:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333111296:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019183.079030:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019183.079036:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019183/real 1734019183] req@ffff8800a705c000 x1818247333111296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019183.079046:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333111296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019183.079054:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019183.079057:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019183.079060:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333111296:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019183.079066:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333111424:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019183.079081:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019183.079087:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019183/real 1734019183] req@ffff8800a705e300 x1818247333111424/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019183.079097:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333111424/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019183.079105:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019183.079107:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019183.079110:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333111424:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019183.079116:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333111552:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019183.079128:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019183.079134:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019183/real 1734019183] req@ffff8800a705d880 x1818247333111552/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019183.079144:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333111552/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019183.079151:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019183.079153:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019183.079156:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333111552:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019184.070496:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019185.072497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019186.074483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019187.076466:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019188.078518:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019188.086341:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019188.086351:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019188.086357:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019188.086370:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3450 00000100:00080000:0.0:1734019188.086383:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019188.086392:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019188.086441:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333111680/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019188.086486:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019188.086491:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019188.086493:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019188.086498:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3450 00000100:00080000:0.0:1734019188.086502:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019188.086506:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019188.086518:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019188.086530:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333111808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019188.086541:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019188.086545:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019188.086547:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019188.086551:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3450 00000100:00080000:0.0:1734019188.086553:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019188.086557:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019188.086563:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019188.086576:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333111936/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019188.086585:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019188.086589:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019188.086590:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019188.086594:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3450 00000100:00080000:0.0:1734019188.086597:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019188.086601:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019188.086606:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019188.086614:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333112064/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019188.086624:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019188.086627:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1734019188.086627:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333111680:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019188.086629:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019188.086633:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3450 00000100:00080000:0.0:1734019188.086635:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019188.086639:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019188.086644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019188.086653:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333112192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019188.086661:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019188.086664:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019188.086667:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019188.086671:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3450 00000100:00080000:0.0:1734019188.086674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019188.086678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019188.086682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019188.086691:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333112320/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019188.086699:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019188.086702:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019188.086703:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1734019188.086704:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019188.086707:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3450 00000100:00080000:0.0:1734019188.086710:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019188.086714:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019188.086718:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019188.086726:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333112448/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019188.086729:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019188/real 1734019188] req@ffff8800a705d180 x1818247333111680/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019188.086758:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333111680/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019188.086770:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019188.086774:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019188.086778:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333111680:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019188.086827:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333111808:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019188.086850:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019188.086859:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019188/real 1734019188] req@ffff8800a705c380 x1818247333111808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019188.086871:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333111808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019188.086879:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019188.086882:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019188.086886:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333111808:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019188.086893:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333111936:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019188.086907:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019188.086913:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019188/real 1734019188] req@ffff8800a705d880 x1818247333111936/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019188.086923:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333111936/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019188.086931:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019188.086934:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019188.086937:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333111936:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019188.086964:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333112064:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019188.086977:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019188.086983:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019188/real 1734019188] req@ffff8800a705e300 x1818247333112064/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019188.086994:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333112064/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019188.087001:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019188.087003:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019188.087006:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333112064:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019188.087012:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333112192:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019188.087024:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019188.087030:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019188/real 1734019188] req@ffff8800a705c000 x1818247333112192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019188.087040:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333112192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019188.087047:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019188.087050:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019188.087053:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333112192:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019188.087059:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333112320:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019188.087074:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019188.087080:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019188/real 1734019188] req@ffff8800a705e680 x1818247333112320/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019188.087090:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333112320/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019188.087097:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019188.087099:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019188.087102:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333112320:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019188.087108:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333112448:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019188.087120:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019188.087125:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019188/real 1734019188] req@ffff8800a705c700 x1818247333112448/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019188.087135:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333112448/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019188.087142:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019188.087144:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019188.087147:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333112448:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019189.080476:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019190.082516:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019191.084502:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019192.086474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019193.088531:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019193.094333:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019193.094344:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019193.094350:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019193.094364:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3455 00000100:00080000:0.0:1734019193.094375:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019193.094384:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019193.094433:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333112576/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019193.094481:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019193.094486:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019193.094488:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019193.094493:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3455 00000100:00080000:0.0:1734019193.094497:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019193.094501:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019193.094513:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019193.094526:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333112704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019193.094536:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019193.094539:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019193.094541:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019193.094545:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3455 00000100:00080000:0.0:1734019193.094548:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019193.094551:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019193.094556:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019193.094565:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333112832/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019193.094574:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019193.094577:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019193.094578:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019193.094582:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3455 00000100:00080000:0.0:1734019193.094584:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019193.094589:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019193.094594:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019193.094603:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333112960/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019193.094612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019193.094616:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019193.094617:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019193.094617:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333112576:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019193.094621:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3455 00000100:00080000:0.0:1734019193.094624:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019193.094628:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019193.094632:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019193.094641:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333113088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019193.094650:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019193.094653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019193.094654:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019193.094658:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3455 00000100:00080000:0.0:1734019193.094661:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019193.094664:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019193.094669:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019193.094678:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333113216/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019193.094686:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019193.094690:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019193.094692:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1734019193.094695:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019193.094696:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3455 00000100:00080000:0.0:1734019193.094699:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019193.094736:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019193.094741:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019193.094753:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333113344/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019193.094764:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019193/real 1734019193] req@ffff8800a705f800 x1818247333112576/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019193.094793:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333112576/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019193.094805:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019193.094808:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019193.094813:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333112576:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019193.094822:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333112704:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019193.094843:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019193.094851:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019193/real 1734019193] req@ffff8800a705d500 x1818247333112704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019193.094861:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333112704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019193.094869:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019193.094872:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019193.094875:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333112704:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019193.094882:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333112832:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019193.094896:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019193.094902:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019193/real 1734019193] req@ffff8800a705ed80 x1818247333112832/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019193.094912:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333112832/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019193.094920:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019193.094929:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019193.094932:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333112832:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019193.094959:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333112960:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019193.094973:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019193.094980:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019193/real 1734019193] req@ffff8800a705ea00 x1818247333112960/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019193.094991:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333112960/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019193.094999:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019193.095001:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019193.095004:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333112960:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019193.095009:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333113088:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019193.095022:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019193.095027:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019193/real 1734019193] req@ffff8800a705fb80 x1818247333113088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019193.095037:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333113088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019193.095045:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019193.095048:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019193.095051:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333113088:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019193.095057:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333113216:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019193.095072:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019193.095078:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019193/real 1734019193] req@ffff8800a705dc00 x1818247333113216/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019193.095089:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333113216/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019193.095096:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019193.095098:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019193.095101:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333113216:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019193.095106:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333113344:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019193.095118:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019193.095124:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019193/real 1734019193] req@ffff8800a705f100 x1818247333113344/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019193.095134:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333113344/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019193.095142:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019193.095144:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019193.095147:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333113344:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019194.090502:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019195.092504:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019196.094470:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019197.096469:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019198.098463:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019198.102324:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019198.102336:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019198.102341:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019198.102356:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3460 00000100:00080000:0.0:1734019198.102367:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019198.102376:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019198.102425:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333113472/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019198.102464:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019198.102469:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019198.102471:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019198.102476:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3460 00000100:00080000:0.0:1734019198.102480:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019198.102484:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019198.102496:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019198.102508:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333113600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019198.102519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019198.102523:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019198.102524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019198.102528:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3460 00000100:00080000:0.0:1734019198.102531:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019198.102535:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019198.102540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019198.102551:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333113728/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019198.102560:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019198.102563:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019198.102565:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019198.102568:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3460 00000100:00080000:0.0:1734019198.102571:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019198.102574:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019198.102579:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019198.102588:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333113856/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019198.102598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019198.102601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019198.102603:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019198.102607:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3460 00000100:00100000:2.0:1734019198.102609:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333113472:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019198.102610:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019198.102614:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019198.102668:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019198.102679:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333113984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019198.102689:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019198.102692:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019198.102694:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019198.102698:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3460 00000100:00080000:0.0:1734019198.102701:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019198.102705:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019198.102709:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019198.102717:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333114112/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019198.102726:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1734019198.102727:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019198.102729:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019198.102731:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019198.102734:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3460 00000100:00080000:0.0:1734019198.102737:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019198.102741:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019198.102745:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019198.102753:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019198/real 1734019198] req@ffff8800a705df80 x1818247333113472/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019198.102754:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333114240/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019198.102781:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333113472/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019198.102793:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019198.102797:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019198.102802:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333113472:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019198.102811:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333113600:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019198.102829:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019198.102836:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019198/real 1734019198] req@ffff8800a705ca80 x1818247333113600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019198.102847:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333113600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019198.102854:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019198.102857:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019198.102861:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333113600:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019198.102867:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333113728:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019198.102881:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019198.102886:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019198/real 1734019198] req@ffff8800a705f100 x1818247333113728/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019198.102897:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333113728/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019198.102904:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019198.102907:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019198.102910:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333113728:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019198.102938:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333113856:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019198.102951:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019198.102957:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019198/real 1734019198] req@ffff8800a705dc00 x1818247333113856/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019198.102967:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333113856/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019198.102975:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019198.102977:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019198.102980:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333113856:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019198.102986:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333113984:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019198.102998:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019198.103004:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019198/real 1734019198] req@ffff8800a705fb80 x1818247333113984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019198.103014:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333113984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019198.103021:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019198.103024:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019198.103026:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333113984:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019198.103033:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333114112:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019198.103049:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019198.103055:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019198/real 1734019198] req@ffff8800a705ea00 x1818247333114112/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019198.103065:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333114112/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019198.103072:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019198.103074:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019198.103077:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333114112:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019198.103083:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333114240:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019198.103095:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019198.103101:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019198/real 1734019198] req@ffff8800a705ed80 x1818247333114240/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019198.103111:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333114240/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019198.103119:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019198.103121:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019198.103124:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333114240:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019199.100525:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019200.102462:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019201.104499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019202.106603:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019203.108459:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019203.110356:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019203.110366:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019203.110370:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019203.110381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3465 00000100:00080000:0.0:1734019203.110391:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019203.110399:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019203.110436:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333114368/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019203.110471:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019203.110475:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019203.110477:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019203.110481:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3465 00000100:00080000:0.0:1734019203.110485:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019203.110489:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019203.110499:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019203.110510:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333114496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019203.110520:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019203.110524:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019203.110525:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019203.110529:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3465 00000100:00080000:0.0:1734019203.110563:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019203.110567:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019203.110574:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019203.110583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333114624/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019203.110592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019203.110596:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019203.110598:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019203.110601:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3465 00000100:00080000:0.0:1734019203.110603:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019203.110607:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019203.110611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019203.110619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333114752/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019203.110625:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333114368:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019203.110627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019203.110631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019203.110632:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019203.110636:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3465 00000100:00080000:0.0:1734019203.110638:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019203.110642:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019203.110646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019203.110654:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333114880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019203.110663:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019203.110666:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019203.110668:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019203.110672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3465 00000100:00080000:0.0:1734019203.110675:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019203.110678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019203.110682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019203.110689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333115008/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734019203.110693:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019203.110697:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019203.110701:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019203.110702:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019203.110705:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3465 00000100:00080000:0.0:1734019203.110708:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019203.110711:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019203.110715:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019203.110715:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019203/real 1734019203] req@ffff8800a705d500 x1818247333114368/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019203.110724:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333115136/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019203.110743:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333114368/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019203.110754:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019203.110758:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019203.110762:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333114368:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019203.110769:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333114496:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019203.110786:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019203.110793:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019203/real 1734019203] req@ffff8800a705f800 x1818247333114496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019203.110805:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333114496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019203.110812:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019203.110815:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019203.110818:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333114496:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019203.110824:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333114624:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019203.110837:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019203.110843:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019203/real 1734019203] req@ffff8800a705c700 x1818247333114624/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019203.110853:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333114624/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019203.110860:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019203.110864:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019203.110867:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333114624:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019203.110892:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333114752:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019203.110906:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019203.110912:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019203/real 1734019203] req@ffff8800a705e680 x1818247333114752/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019203.110923:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333114752/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019203.110930:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019203.110932:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019203.110935:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333114752:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019203.110940:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333114880:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019203.110952:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019203.110957:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019203/real 1734019203] req@ffff8800a705c000 x1818247333114880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019203.110967:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333114880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019203.110974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019203.110976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019203.110980:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333114880:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019203.110985:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333115008:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019203.111000:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019203.111006:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019203/real 1734019203] req@ffff8800a705e300 x1818247333115008/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019203.111017:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333115008/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019203.111023:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019203.111025:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019203.111028:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333115008:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019203.111033:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333115136:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019203.111045:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019203.111050:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019203/real 1734019203] req@ffff8800a705d880 x1818247333115136/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019203.111060:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333115136/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019203.111067:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019203.111069:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019203.111072:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333115136:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734019204.110546:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019205.112564:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019206.114464:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019207.116434:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019208.118416:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019208.118428:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019208.118432:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019208.118443:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3470 00000100:00080000:0.0:1734019208.118495:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019208.118504:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00010000:00010000:2.0:1734019208.118534:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1734019208.118540:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333115264/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019208.118586:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019208.118590:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019208.118592:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019208.118597:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3470 00000100:00080000:0.0:1734019208.118600:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019208.118604:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019208.118613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019208.118618:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333115264:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019208.118624:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333115392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019208.118634:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019208.118637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019208.118639:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019208.118643:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3470 00000100:00080000:0.0:1734019208.118645:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019208.118649:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019208.118654:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019208.118664:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333115520/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019208.118673:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019208.118676:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019208.118677:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019208.118681:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3470 00000100:00080000:0.0:1734019208.118684:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734019208.118686:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019208.118688:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019208.118693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019208.118701:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333115648/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019208.118709:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734019208.118709:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019208/real 1734019208] req@ffff8800a705c380 x1818247333115264/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019208.118712:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019208.118713:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019208.118717:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3470 00000100:00080000:0.0:1734019208.118719:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019208.118723:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019208.118727:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019208.118729:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333115264/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019208.118735:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333115776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019208.118740:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019208.118743:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019208.118743:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:0.0:1734019208.118746:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019208.118747:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019208.118748:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333115264:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019208.118751:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3470 00000100:00080000:0.0:1734019208.118754:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019208.118757:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019208.118761:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019208.118768:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333115904/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019208.118770:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333115392:192.168.201.152@tcp:38:kworker.0 00000100:00080000:0.0:1734019208.118776:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019208.118780:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019208.118782:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019208.118785:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3470 00000100:00080000:0.0:1734019208.118788:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734019208.118788:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019208.118792:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019208.118796:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019208.118796:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019208/real 1734019208] req@ffff8800a705d180 x1818247333115392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019208.118803:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333116032/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019208.118807:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333115392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019208.118815:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019208.118817:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019208.118821:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333115392:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019208.118827:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333115520:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019208.118840:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019208.118846:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019208/real 1734019208] req@ffff8800a705d880 x1818247333115520/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019208.118856:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333115520/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019208.118872:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019208.118874:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019208.118877:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333115520:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019208.118883:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333115648:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019208.118895:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019208.118900:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019208/real 1734019208] req@ffff8800a705e300 x1818247333115648/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019208.118910:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333115648/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019208.118917:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019208.118919:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019208.118923:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333115648:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019208.118928:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333115776:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019208.118940:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019208.118946:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019208/real 1734019208] req@ffff8800a705c000 x1818247333115776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019208.118955:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333115776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019208.118963:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019208.118965:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019208.118968:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333115776:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019208.118984:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333115904:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019208.118995:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019208.119001:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019208/real 1734019208] req@ffff8800a705e680 x1818247333115904/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019208.119011:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333115904/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019208.119018:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019208.119021:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019208.119024:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333115904:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019208.119029:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333116032:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019208.119040:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019208.119046:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019208/real 1734019208] req@ffff8800a705c700 x1818247333116032/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019208.119056:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333116032/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019208.119063:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019208.119065:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019208.119068:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333116032:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019209.120516:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019210.122616:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019211.124505:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019212.126594:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019213.126335:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019213.126345:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019213.126350:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019213.126411:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3475 00000100:00080000:0.0:1734019213.126422:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019213.126431:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019213.126467:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333116160/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019213.126501:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019213.126506:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019213.126508:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019213.126512:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3475 00000100:00080000:0.0:1734019213.126516:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019213.126520:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019213.126530:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019213.126540:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333116288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019213.126551:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019213.126554:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019213.126556:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019213.126559:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3475 00000100:00080000:0.0:1734019213.126562:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019213.126565:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019213.126570:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019213.126578:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333116416/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019213.126587:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019213.126590:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019213.126591:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019213.126595:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3475 00000100:00080000:0.0:1734019213.126597:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019213.126601:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019213.126605:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019213.126612:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333116544/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019213.126619:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333116160:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019213.126621:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019213.126624:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019213.126626:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019213.126629:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3475 00000100:00080000:0.0:1734019213.126632:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019213.126636:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019213.126640:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019213.126647:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333116672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019213.126655:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019213.126658:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019213.126660:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019213.126663:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3475 00000100:00080000:0.0:1734019213.126666:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019213.126669:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019213.126673:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019213.126680:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333116800/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734019213.126687:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019213.126688:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019213.126691:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019213.126693:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019213.126696:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3475 00000100:00080000:0.0:1734019213.126699:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019213.126702:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019213.126706:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019213.126710:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019213/real 1734019213] req@ffff8800a705f800 x1818247333116160/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019213.126713:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333116928/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019213.126729:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333116160/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019213.126740:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019213.126744:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019213.126749:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333116160:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019213.126756:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333116288:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019213.126772:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019213.126779:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019213/real 1734019213] req@ffff8800a705d500 x1818247333116288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019213.126790:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333116288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019213.126797:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019213.126800:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019213.126803:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333116288:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019213.126809:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333116416:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019213.126822:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019213.126827:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019213/real 1734019213] req@ffff8800a705ed80 x1818247333116416/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019213.126837:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333116416/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019213.126853:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019213.126856:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019213.126859:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333116416:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019213.126884:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333116544:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019213.126896:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019213.126903:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019213/real 1734019213] req@ffff8800a705ea00 x1818247333116544/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019213.126913:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333116544/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019213.126922:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019213.126924:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019213.126927:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333116544:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019213.126932:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333116672:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019213.126943:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019213.126949:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019213/real 1734019213] req@ffff8800a705fb80 x1818247333116672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019213.126958:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333116672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019213.126966:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019213.126969:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019213.126972:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333116672:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019213.126978:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333116800:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019213.126990:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019213.126995:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019213/real 1734019213] req@ffff8800a705dc00 x1818247333116800/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019213.127005:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333116800/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019213.127012:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019213.127014:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019213.127017:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333116800:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019213.127022:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333116928:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019213.127033:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019213.127039:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019213/real 1734019213] req@ffff8800a705f100 x1818247333116928/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019213.127049:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333116928/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019213.127056:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019213.127058:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019213.127061:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333116928:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019213.128467:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019214.130583:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019215.132527:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019216.134507:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019217.136515:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019218.134367:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019218.134373:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019218.134376:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019218.134384:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3480 00000100:00080000:0.0:1734019218.134410:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019218.134418:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019218.134456:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333117056/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019218.134494:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019218.134500:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019218.134504:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019218.134510:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3480 00000100:00080000:0.0:1734019218.134515:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019218.134521:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019218.134533:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019218.134551:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333117184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019218.134567:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019218.134571:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019218.134574:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019218.134577:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333117056:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019218.134579:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3480 00000100:00080000:0.0:1734019218.134583:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019218.134588:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019218.134596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019218.134610:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333117312/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019218.134622:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019218.134627:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019218.134629:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019218.134634:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3480 00000100:00080000:0.0:1734019218.134637:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019218.134641:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019218.134647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019218.134650:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019218.134660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333117440/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019218.134673:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734019218.134674:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019218/real 1734019218] req@ffff8800a705ca80 x1818247333117056/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019218.134678:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019218.134680:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019218.134685:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3480 00000100:00080000:0.0:1734019218.134690:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019218.134694:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:2.0:1734019218.134699:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333117056/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734019218.134701:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019218.134713:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333117568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019218.134713:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019218.134718:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019218.134724:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333117056:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019218.134727:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019218.134732:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019218.134734:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019218.134740:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3480 00000100:00080000:0.0:1734019218.134744:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019218.134748:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:2.0:1734019218.134751:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333117184:192.168.201.152@tcp:38:kworker.0 00000008:00100000:0.0:1734019218.134755:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019218.134766:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333117696/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734019218.134777:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019218.134782:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019218.134787:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1734019218.134788:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019218/real 1734019218] req@ffff8800a705df80 x1818247333117184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019218.134790:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019218.134795:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3480 00000100:00080000:0.0:1734019218.134800:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019218.134805:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:2.0:1734019218.134805:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333117184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734019218.134812:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1734019218.134819:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019218.134823:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019218.134824:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333117824/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019218.134828:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333117184:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019218.134837:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333117312:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019218.134860:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019218.134871:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019218/real 1734019218] req@ffff8800a705f100 x1818247333117312/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019218.134892:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333117312/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019218.134922:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019218.134930:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019218.134935:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333117312:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019218.134942:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333117440:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019218.134963:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019218.134971:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019218/real 1734019218] req@ffff8800a705dc00 x1818247333117440/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019218.134983:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333117440/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019218.134991:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019218.134993:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019218.134996:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333117440:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019218.135001:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333117568:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019218.135013:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019218.135018:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019218/real 1734019218] req@ffff8800a705fb80 x1818247333117568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019218.135028:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333117568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019218.135036:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019218.135038:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019218.135041:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333117568:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019218.135060:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333117696:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019218.135073:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019218.135079:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019218/real 1734019218] req@ffff8800a705ea00 x1818247333117696/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019218.135089:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333117696/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019218.135096:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019218.135098:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019218.135101:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333117696:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019218.135106:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333117824:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019218.135117:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019218.135122:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019218/real 1734019218] req@ffff8800a705ed80 x1818247333117824/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019218.135132:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333117824/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019218.135139:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019218.135141:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019218.135144:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333117824:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734019218.138426:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019219.140512:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019220.142511:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019221.144448:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019222.146505:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019223.142375:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019223.142390:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019223.142396:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019223.142411:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3485 00000100:00080000:0.0:1734019223.142424:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019223.142434:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019223.142483:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333117952/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019223.142532:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019223.142536:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019223.142538:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019223.142544:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3485 00000100:00080000:0.0:1734019223.142548:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019223.142552:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019223.142564:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019223.142576:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333118080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019223.142586:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019223.142589:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019223.142591:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019223.142595:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3485 00000100:00080000:0.0:1734019223.142598:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019223.142601:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019223.142606:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019223.142615:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333118208/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019223.142624:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019223.142627:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019223.142629:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019223.142632:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3485 00000100:00080000:0.0:1734019223.142635:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019223.142639:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019223.142644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019223.142652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333118336/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019223.142662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019223.142665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019223.142667:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019223.142671:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3485 00000100:00080000:0.0:1734019223.142673:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734019223.142675:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333117952:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019223.142677:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019223.142682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019223.142690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333118464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019223.142699:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019223.142702:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019223.142703:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019223.142708:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3485 00000100:00080000:0.0:1734019223.142710:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019223.142714:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019223.142719:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019223.142728:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333118592/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019223.142736:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019223.142740:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019223.142742:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019223.142745:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3485 00000100:00080000:0.0:1734019223.142749:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019223.142753:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:2.0:1734019223.142753:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734019223.142757:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019223.142768:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333118720/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019223.142777:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019223/real 1734019223] req@ffff8800a705d500 x1818247333117952/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019223.142797:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333117952/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019223.142809:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019223.142812:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019223.142817:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333117952:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019223.142826:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333118080:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019223.142844:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019223.142851:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019223/real 1734019223] req@ffff8800a705f800 x1818247333118080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019223.142862:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333118080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019223.142870:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019223.142873:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019223.142876:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333118080:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019223.142883:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333118208:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019223.142896:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019223.142902:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019223/real 1734019223] req@ffff8800a705c700 x1818247333118208/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019223.142913:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333118208/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019223.142922:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019223.142925:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019223.142928:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333118208:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019223.142954:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333118336:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019223.142968:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019223.142974:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019223/real 1734019223] req@ffff8800a705e680 x1818247333118336/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019223.142984:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333118336/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019223.142992:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019223.142995:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019223.142998:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333118336:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019223.143003:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333118464:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019223.143015:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019223.143021:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019223/real 1734019223] req@ffff8800a705c000 x1818247333118464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019223.143031:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333118464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019223.143038:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019223.143041:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019223.143044:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333118464:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019223.143051:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333118592:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019223.143063:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019223.143069:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019223/real 1734019223] req@ffff8800a705e300 x1818247333118592/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019223.143079:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333118592/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019223.143087:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019223.143089:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019223.143092:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333118592:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019223.143098:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333118720:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019223.143110:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019223.143115:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019223/real 1734019223] req@ffff8800a705d880 x1818247333118720/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019223.143125:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333118720/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019223.143132:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019223.143135:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019223.143138:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333118720:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019223.148461:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019224.150477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019225.152481:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019226.154508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019227.156471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019228.150353:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019228.150364:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019228.150368:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019228.150379:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3490 00000100:00080000:0.0:1734019228.150390:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019228.150398:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019228.150435:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333118848/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019228.150470:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019228.150474:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019228.150477:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019228.150481:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3490 00000100:00080000:0.0:1734019228.150484:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019228.150488:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019228.150498:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019228.150509:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333118976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019228.150520:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019228.150524:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019228.150525:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019228.150529:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3490 00000100:00080000:0.0:1734019228.150531:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019228.150535:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019228.150540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019228.150550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333119104/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019228.150558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019228.150561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019228.150563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019228.150566:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3490 00000100:00080000:0.0:1734019228.150568:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019228.150572:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019228.150576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019228.150584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333119232/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019228.150593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734019228.150596:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333118848:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019228.150597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019228.150598:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019228.150602:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3490 00000100:00080000:0.0:1734019228.150606:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019228.150609:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019228.150614:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019228.150622:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333119360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019228.150630:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019228.150633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019228.150634:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019228.150637:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3490 00000100:00080000:0.0:1734019228.150640:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019228.150643:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019228.150648:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019228.150655:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333119488/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019228.150663:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019228.150666:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019228.150667:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019228.150670:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3490 00000100:00080000:0.0:1734019228.150673:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019228.150676:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019228.150680:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019228.150688:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333119616/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019228.150692:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333118976:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019228.150716:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333119104:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019228.150740:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333119232:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019228.150755:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333119360:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019228.150769:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333119488:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019228.150783:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333119616:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734019228.158487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019229.160444:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019230.162586:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1734019231.156546:0:1956:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:1.0:1734019231.156598:0:1956:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1734019231.156623:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019231.156666:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019231.156675:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019231.156684:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019231.156692:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019231.156700:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019231.156707:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:2.0:1734019231.156761:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019228/real 1734019231] req@ffff8800a705e300 x1818247333119232/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019231.156795:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333119232/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019231.156809:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019231.156814:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019231.156820:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333119232:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019231.156828:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019228/real 1734019231] req@ffff8800a705c000 x1818247333119360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019231.156840:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333119360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019231.156848:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019231.156851:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019231.156855:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333119360:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019231.156863:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019228/real 1734019231] req@ffff8800a705e680 x1818247333119488/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019231.156875:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333119488/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019231.156883:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019231.156886:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019231.156889:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333119488:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019231.156895:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019228/real 1734019231] req@ffff8800a705c700 x1818247333119616/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019231.156906:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333119616/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019231.156923:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019231.156926:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019231.156929:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333119616:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019231.156935:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019228/real 1734019231] req@ffff8800a705d180 x1818247333118848/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019231.156948:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333118848/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019231.156955:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019231.156958:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019231.156962:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333118848:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019231.156968:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019228/real 1734019231] req@ffff8800a705c380 x1818247333118976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019231.156980:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333118976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019231.156988:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019231.156992:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019231.156995:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333118976:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019231.157001:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019228/real 1734019231] req@ffff8800a705d880 x1818247333119104/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019231.157012:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333119104/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019231.157019:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019231.157022:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019231.157025:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333119104:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019231.164511:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019232.166567:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019233.158391:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019233.158403:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019233.158408:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019233.158422:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3495 00000100:00080000:0.0:1734019233.158435:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019233.158445:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019233.158494:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333119744/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019233.158531:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019233.158535:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019233.158538:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019233.158543:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3495 00000100:00080000:0.0:1734019233.158547:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019233.158551:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019233.158563:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019233.158575:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333119872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019233.158585:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019233.158589:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019233.158591:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019233.158595:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3495 00000100:00080000:0.0:1734019233.158598:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019233.158601:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019233.158607:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019233.158616:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333120000/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019233.158625:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019233.158628:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019233.158630:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019233.158633:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3495 00000100:00080000:0.0:1734019233.158636:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019233.158639:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019233.158644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019233.158646:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333119744:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019233.158653:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333120128/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019233.158661:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019233.158665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019233.158666:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019233.158670:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3495 00000100:00080000:0.0:1734019233.158673:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019233.158676:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019233.158681:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019233.158689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333120256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019233.158698:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019233.158701:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019233.158703:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019233.158708:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3495 00000100:00080000:0.0:1734019233.158711:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019233.158714:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019233.158719:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019233.158727:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333120384/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734019233.158732:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019233.158735:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019233.158738:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019233.158740:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019233.158743:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3495 00000100:00080000:0.0:1734019233.158746:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019233.158749:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019233.158753:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019233.158758:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019233/real 1734019233] req@ffff8800a705f800 x1818247333119744/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019233.158764:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333120512/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019233.158779:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333119744/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019233.158791:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019233.158794:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019233.158801:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333119744:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019233.158808:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333119872:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019233.158828:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019233.158835:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019233/real 1734019233] req@ffff8800a705d500 x1818247333119872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019233.158845:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333119872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019233.158854:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019233.158857:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019233.158860:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333119872:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019233.158866:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333120000:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019233.158881:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019233.158886:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019233/real 1734019233] req@ffff8800a705ed80 x1818247333120000/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019233.158897:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333120000/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019233.158906:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019233.158909:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019233.158912:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333120000:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019233.158941:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333120128:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019233.158953:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019233.158961:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019233/real 1734019233] req@ffff8800a705ea00 x1818247333120128/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019233.158971:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333120128/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019233.158979:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019233.158981:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019233.158984:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333120128:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019233.158989:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333120256:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019233.159013:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019233.159019:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019233/real 1734019233] req@ffff8800a705fb80 x1818247333120256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019233.159070:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333120256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019233.159078:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019233.159080:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019233.159083:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333120256:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019233.159088:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333120384:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019233.159104:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019233.159111:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019233/real 1734019233] req@ffff8800a705dc00 x1818247333120384/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019233.159121:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333120384/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019233.159137:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019233.159140:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019233.159143:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333120384:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019233.159148:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333120512:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019233.159160:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019233.159166:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019233/real 1734019233] req@ffff8800a705f100 x1818247333120512/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019233.159176:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333120512/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019233.159183:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019233.159186:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019233.159189:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333120512:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019233.168652:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019234.170576:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019235.172498:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019236.174540:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019237.176527:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019238.166395:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019238.166406:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019238.166411:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019238.166422:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3500 00000100:00080000:0.0:1734019238.166433:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019238.166441:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019238.166479:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333120640/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019238.166523:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019238.166527:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019238.166529:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019238.166534:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3500 00000100:00080000:0.0:1734019238.166537:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019238.166541:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019238.166551:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019238.166562:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333120768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019238.166571:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019238.166575:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019238.166576:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019238.166580:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3500 00000100:00080000:0.0:1734019238.166582:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019238.166586:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019238.166590:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019238.166600:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333120896/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019238.166608:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019238.166611:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019238.166613:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019238.166616:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3500 00000100:00080000:0.0:1734019238.166619:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019238.166623:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019238.166627:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019238.166635:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333121024/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019238.166644:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734019238.166645:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333120640:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019238.166647:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019238.166649:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019238.166652:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3500 00000100:00080000:0.0:1734019238.166655:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019238.166658:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019238.166663:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019238.166670:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333121152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019238.166678:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019238.166681:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019238.166683:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019238.166686:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3500 00000100:00080000:0.0:1734019238.166688:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019238.166692:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019238.166696:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019238.166703:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333121280/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734019238.166711:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019238.166712:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019238.166715:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019238.166717:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019238.166720:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3500 00000100:00080000:0.0:1734019238.166723:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019238.166726:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019238.166731:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019238.166734:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019238/real 1734019238] req@ffff8800a705df80 x1818247333120640/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019238.166738:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333121408/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019238.166752:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333120640/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019238.166763:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019238.166767:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019238.166771:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333120640:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019238.166780:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333120768:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019238.166796:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019238.166803:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019238/real 1734019238] req@ffff8800a705ca80 x1818247333120768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019238.166814:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333120768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019238.166822:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019238.166825:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019238.166828:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333120768:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019238.166833:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333120896:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019238.166846:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019238.166852:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019238/real 1734019238] req@ffff8800a705f100 x1818247333120896/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019238.166862:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333120896/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019238.166869:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019238.166872:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019238.166874:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333120896:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019238.166900:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333121024:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019238.166912:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019238.166918:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019238/real 1734019238] req@ffff8800a705dc00 x1818247333121024/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019238.166929:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333121024/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019238.166983:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019238.166986:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019238.166990:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333121024:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019238.166996:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333121152:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019238.167011:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019238.167018:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019238/real 1734019238] req@ffff8800a705fb80 x1818247333121152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019238.167029:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333121152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019238.167037:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019238.167039:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019238.167042:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333121152:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019238.167047:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333121280:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019238.167059:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019238.167064:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019238/real 1734019238] req@ffff8800a705ea00 x1818247333121280/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019238.167074:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333121280/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019238.167081:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019238.167084:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019238.167087:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333121280:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019238.167093:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333121408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019238.167104:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019238.167110:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019238/real 1734019238] req@ffff8800a705ed80 x1818247333121408/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019238.167120:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333121408/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019238.167127:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019238.167129:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019238.167132:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333121408:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019238.178486:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019239.180477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019240.182473:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019241.184495:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019242.186529:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019243.174358:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019243.174369:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019243.174373:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019243.174384:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3505 00000100:00080000:0.0:1734019243.174394:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019243.174401:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019243.174436:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333121536/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019243.174473:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019243.174478:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019243.174480:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019243.174484:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3505 00000100:00080000:0.0:1734019243.174487:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019243.174491:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019243.174501:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019243.174512:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333121664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019243.174523:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019243.174526:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019243.174528:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019243.174532:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3505 00000100:00080000:0.0:1734019243.174534:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019243.174538:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019243.174542:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019243.174550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333121792/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019243.174558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019243.174561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019243.174563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019243.174566:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3505 00000100:00080000:0.0:1734019243.174569:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019243.174572:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019243.174576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019243.174584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333121920/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019243.174592:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333121536:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019243.174593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019243.174597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019243.174598:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019243.174602:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3505 00000100:00080000:0.0:1734019243.174606:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019243.174609:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019243.174613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019243.174621:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333122048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019243.174629:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019243.174632:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019243.174633:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019243.174637:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3505 00000100:00080000:0.0:1734019243.174640:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019243.174643:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019243.174647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019243.174654:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333122176/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734019243.174658:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019243.174662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019243.174665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019243.174667:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019243.174670:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3505 00000100:00080000:0.0:1734019243.174673:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019243.174676:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019243.174680:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019243.174680:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019243/real 1734019243] req@ffff8800a705d500 x1818247333121536/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019243.174689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333122304/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019243.174700:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333121536/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019243.174710:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019243.174713:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019243.174718:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333121536:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019243.174725:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333121664:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019243.174745:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019243.174751:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019243/real 1734019243] req@ffff8800a705f800 x1818247333121664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019243.174762:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333121664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019243.174770:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019243.174773:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019243.174776:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333121664:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019243.174782:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333121792:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019243.174795:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019243.174801:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019243/real 1734019243] req@ffff8800a705d880 x1818247333121792/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019243.174811:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333121792/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019243.174818:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019243.174821:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019243.174824:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333121792:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019243.174848:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333121920:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019243.174913:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019243.174922:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019243/real 1734019243] req@ffff8800a705c380 x1818247333121920/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019243.174939:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333121920/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019243.174947:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019243.174950:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019243.174953:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333121920:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019243.174958:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333122048:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019243.174969:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019243.174975:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019243/real 1734019243] req@ffff8800a705d180 x1818247333122048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019243.174985:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333122048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019243.174992:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019243.174995:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019243.174998:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333122048:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019243.175002:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333122176:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019243.175014:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019243.175019:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019243/real 1734019243] req@ffff8800a705c700 x1818247333122176/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019243.175029:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333122176/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019243.175036:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019243.175039:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019243.175042:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333122176:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019243.175047:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333122304:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019243.175059:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019243.175065:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019243/real 1734019243] req@ffff8800a705e680 x1818247333122304/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019243.175074:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333122304/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019243.175082:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019243.175084:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019243.175087:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333122304:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019243.188457:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019244.190473:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019245.192452:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019246.194467:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019247.196375:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019248.182336:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019248.182343:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019248.182346:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019248.182354:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3510 00000100:00080000:0.0:1734019248.182361:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019248.182366:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019248.182391:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333122432/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019248.182413:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019248.182416:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019248.182417:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019248.182420:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3510 00000100:00080000:0.0:1734019248.182421:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019248.182423:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019248.182430:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019248.182435:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333122560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019248.182441:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019248.182443:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019248.182444:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019248.182446:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3510 00000100:00080000:0.0:1734019248.182447:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019248.182449:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019248.182452:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019248.182458:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333122688/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019248.182463:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019248.182465:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019248.182465:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019248.182467:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3510 00000100:00080000:0.0:1734019248.182469:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019248.182470:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019248.182473:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019248.182477:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333122816/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019248.182481:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019248.182483:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019248.182483:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019248.182485:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3510 00000100:00080000:0.0:1734019248.182487:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019248.182489:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019248.182491:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019248.182491:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333122432:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019248.182495:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333122944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019248.182500:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019248.182501:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019248.182503:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019248.182505:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3510 00000100:00080000:0.0:1734019248.182506:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019248.182508:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019248.182510:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019248.182514:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333123072/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019248.182519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019248.182520:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019248.182521:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019248.182523:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3510 00000100:00080000:0.0:1734019248.182524:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019248.182526:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019248.182528:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019248.182532:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333123200/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734019248.182536:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019248.182550:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019248/real 1734019248] req@ffff8800a705c000 x1818247333122432/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019248.182560:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333122432/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019248.182566:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019248.182568:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019248.182570:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333122432:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019248.182574:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333122560:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019248.182585:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019248.182589:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019248/real 1734019248] req@ffff8800a705e300 x1818247333122560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019248.182595:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333122560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019248.182600:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019248.182601:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019248.182603:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333122560:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019248.182605:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333122688:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019248.182612:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019248.182615:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019248/real 1734019248] req@ffff8800a705e680 x1818247333122688/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019248.182620:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333122688/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019248.182624:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019248.182625:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019248.182626:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333122688:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019248.182629:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333122816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019248.182636:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019248.182639:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019248/real 1734019248] req@ffff8800a705c700 x1818247333122816/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019248.182648:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333122816/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019248.182652:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019248.182653:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019248.182654:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333122816:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019248.182669:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333122944:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019248.182675:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019248.182678:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019248/real 1734019248] req@ffff8800a705d180 x1818247333122944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019248.182683:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333122944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019248.182687:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019248.182688:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019248.182690:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333122944:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019248.182692:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333123072:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019248.182699:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019248.182701:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019248/real 1734019248] req@ffff8800a705c380 x1818247333123072/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019248.182706:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333123072/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019248.182710:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019248.182712:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019248.182713:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333123072:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019248.182717:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333123200:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019248.182723:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019248.182726:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019248/real 1734019248] req@ffff8800a705d880 x1818247333123200/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019248.182731:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333123200/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019248.182734:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019248.182735:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019248.182737:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333123200:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019248.198345:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019249.200464:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019250.202468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019251.204456:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019252.206407:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019253.190356:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019253.190365:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019253.190370:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019253.190381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3515 00000100:00080000:0.0:1734019253.190390:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019253.190398:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019253.190435:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333123328/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019253.190477:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019253.190482:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019253.190484:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019253.190488:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3515 00000100:00080000:0.0:1734019253.190492:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019253.190496:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019253.190506:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019253.190517:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333123456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019253.190526:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019253.190530:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019253.190531:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019253.190535:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3515 00000100:00080000:0.0:1734019253.190538:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019253.190541:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019253.190546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019253.190554:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333123584/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019253.190562:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019253.190565:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019253.190567:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019253.190570:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3515 00000100:00080000:0.0:1734019253.190572:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019253.190576:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019253.190581:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019253.190589:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333123712/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019253.190596:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333123328:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019253.190598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019253.190601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019253.190602:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019253.190606:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3515 00000100:00080000:0.0:1734019253.190608:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019253.190611:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019253.190616:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019253.190623:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333123840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019253.190631:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019253.190634:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019253.190636:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019253.190639:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3515 00000100:00080000:0.0:1734019253.190642:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019253.190645:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019253.190649:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019253.190657:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333123968/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734019253.190663:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019253.190665:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019253.190669:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019253.190670:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019253.190674:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3515 00000100:00080000:0.0:1734019253.190677:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734019253.190719:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019253/real 1734019253] req@ffff8800a705f800 x1818247333123328/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019253.190725:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019253.190731:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019253.190739:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333123328/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019253.190741:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333124096/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019253.190751:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019253.190754:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019253.190759:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333123328:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019253.190766:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333123456:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019253.190790:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019253.190797:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019253/real 1734019253] req@ffff8800a705d500 x1818247333123456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019253.190809:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333123456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019253.190817:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019253.190819:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019253.190822:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333123456:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019253.190828:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333123584:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019253.190842:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019253.190848:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019253/real 1734019253] req@ffff8800a705ed80 x1818247333123584/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019253.190859:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333123584/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019253.190867:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019253.190869:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019253.190872:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333123584:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019253.190896:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333123712:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019253.190908:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019253.190915:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019253/real 1734019253] req@ffff8800a705ea00 x1818247333123712/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019253.190931:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333123712/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019253.190939:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019253.190941:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019253.190944:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333123712:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019253.190949:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333123840:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019253.190960:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019253.190966:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019253/real 1734019253] req@ffff8800a705fb80 x1818247333123840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019253.190976:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333123840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019253.190983:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019253.190985:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019253.190988:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333123840:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019253.190993:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333123968:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019253.191005:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019253.191011:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019253/real 1734019253] req@ffff8800a705dc00 x1818247333123968/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019253.191020:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333123968/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019253.191027:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019253.191030:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019253.191034:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333123968:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019253.191039:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333124096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019253.191050:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019253.191056:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019253/real 1734019253] req@ffff8800a705f100 x1818247333124096/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019253.191066:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333124096/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019253.191073:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019253.191075:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019253.191078:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333124096:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019253.208450:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019254.210461:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019255.212478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019256.214477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019257.216492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019258.198361:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019258.198371:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019258.198375:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019258.198385:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3520 00000100:00080000:0.0:1734019258.198395:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019258.198404:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019258.198440:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333124224/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019258.198476:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019258.198480:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019258.198482:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019258.198487:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3520 00000100:00080000:0.0:1734019258.198490:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019258.198494:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019258.198505:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019258.198515:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333124352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019258.198526:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019258.198529:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019258.198531:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019258.198534:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3520 00000100:00080000:0.0:1734019258.198537:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019258.198540:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019258.198545:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019258.198555:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333124480/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019258.198564:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019258.198567:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019258.198568:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019258.198572:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3520 00000100:00080000:0.0:1734019258.198575:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019258.198578:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019258.198582:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019258.198590:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333124608/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019258.198592:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333124224:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019258.198640:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019258.198643:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019258.198646:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019258.198649:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3520 00000100:00080000:0.0:1734019258.198653:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019258.198656:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019258.198661:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019258.198670:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333124736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019258.198679:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019258.198682:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019258.198683:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019258.198687:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3520 00000100:00080000:0.0:1734019258.198690:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019258.198693:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:2.0:1734019258.198693:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:0.0:1734019258.198698:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019258.198705:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333124864/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019258.198713:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734019258.198715:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019258/real 1734019258] req@ffff8800a705ca80 x1818247333124224/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019258.198716:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019258.198717:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019258.198721:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3520 00000100:00080000:0.0:1734019258.198723:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019258.198727:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019258.198731:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019258.198734:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333124224/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019258.198739:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333124992/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019258.198745:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019258.198748:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019258.198753:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333124224:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019258.198760:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333124352:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019258.198780:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019258.198787:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019258/real 1734019258] req@ffff8800a705df80 x1818247333124352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019258.198798:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333124352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019258.198806:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019258.198808:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019258.198812:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333124352:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019258.198817:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333124480:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019258.198830:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019258.198836:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019258/real 1734019258] req@ffff8800a705f100 x1818247333124480/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019258.198846:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333124480/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019258.198854:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019258.198856:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019258.198859:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333124480:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019258.198884:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333124608:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019258.198896:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019258.198902:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019258/real 1734019258] req@ffff8800a705dc00 x1818247333124608/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019258.198920:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333124608/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019258.198927:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019258.198930:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019258.198932:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333124608:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019258.198937:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333124736:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019258.198949:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019258.198954:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019258/real 1734019258] req@ffff8800a705fb80 x1818247333124736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019258.198964:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333124736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019258.198971:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019258.198973:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019258.198976:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333124736:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019258.198981:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333124864:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019258.198993:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019258.198999:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019258/real 1734019258] req@ffff8800a705ea00 x1818247333124864/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019258.199008:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333124864/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019258.199015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019258.199018:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019258.199021:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333124864:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019258.199027:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333124992:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019258.199038:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019258.199044:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019258/real 1734019258] req@ffff8800a705ed80 x1818247333124992/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019258.199053:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333124992/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019258.199060:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019258.199063:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019258.199066:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333124992:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019258.218487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019259.220501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019260.222495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019261.224508:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019262.226481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019263.206370:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019263.206380:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019263.206385:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019263.206395:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3525 00000100:00080000:0.0:1734019263.206406:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019263.206414:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019263.206449:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333125120/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019263.206485:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019263.206489:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019263.206491:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019263.206496:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3525 00000100:00080000:0.0:1734019263.206499:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019263.206503:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019263.206552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019263.206565:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333125248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019263.206576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019263.206579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019263.206581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019263.206585:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3525 00000100:00080000:0.0:1734019263.206587:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019263.206591:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019263.206596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019263.206604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333125376/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019263.206614:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019263.206617:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019263.206619:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019263.206621:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333125120:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019263.206623:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3525 00000100:00080000:0.0:1734019263.206626:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019263.206629:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019263.206633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019263.206641:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333125504/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019263.206649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019263.206652:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019263.206654:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019263.206657:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3525 00000100:00080000:0.0:1734019263.206659:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019263.206662:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019263.206667:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019263.206674:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333125632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019263.206682:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019263.206686:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019263.206688:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1734019263.206689:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019263.206692:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3525 00000100:00080000:0.0:1734019263.206695:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019263.206699:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019263.206703:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019263.206710:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333125760/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019263.206711:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019263/real 1734019263] req@ffff8800a705d500 x1818247333125120/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019263.206718:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019263.206721:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019263.206723:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019263.206726:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3525 00000100:00080000:0.0:1734019263.206729:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734019263.206730:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333125120/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019263.206732:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019263.206736:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1734019263.206741:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019263.206744:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019263.206745:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333125888/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019263.206749:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333125120:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019263.206756:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333125248:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019263.206776:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019263.206782:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019263/real 1734019263] req@ffff8800a705f800 x1818247333125248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019263.206794:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333125248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019263.206802:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019263.206804:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019263.206807:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333125248:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019263.206831:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333125376:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019263.206844:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019263.206850:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019263/real 1734019263] req@ffff8800a705d880 x1818247333125376/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019263.206860:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333125376/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019263.206868:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019263.206870:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019263.206873:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333125376:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019263.206878:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333125504:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019263.206890:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019263.206895:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019263/real 1734019263] req@ffff8800a705c380 x1818247333125504/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019263.206922:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333125504/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019263.206930:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019263.206932:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019263.206935:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333125504:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019263.206940:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333125632:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019263.206951:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019263.206957:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019263/real 1734019263] req@ffff8800a705d180 x1818247333125632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019263.206967:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333125632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019263.206974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019263.206976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019263.206979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333125632:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019263.206984:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333125760:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019263.206996:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019263.207002:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019263/real 1734019263] req@ffff8800a705c700 x1818247333125760/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019263.207012:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333125760/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019263.207019:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019263.207022:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019263.207025:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333125760:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019263.207030:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333125888:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019263.207042:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019263.207047:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019263/real 1734019263] req@ffff8800a705e680 x1818247333125888/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019263.207057:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333125888/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019263.207064:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019263.207066:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019263.207069:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333125888:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019263.228454:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019264.230448:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019265.232428:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019266.234455:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019267.236504:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019268.214352:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019268.214362:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019268.214366:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019268.214376:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3530 00000100:00080000:0.0:1734019268.214385:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019268.214391:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019268.214422:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333126016/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019268.214482:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019268.214485:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019268.214486:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019268.214490:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3530 00000100:00080000:0.0:1734019268.214493:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019268.214496:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019268.214504:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019268.214514:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333126144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019268.214521:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019268.214524:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019268.214525:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019268.214528:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3530 00000100:00080000:0.0:1734019268.214530:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019268.214533:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019268.214536:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019268.214544:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333126272/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019268.214550:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019268.214553:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019268.214554:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019268.214556:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3530 00000100:00080000:0.0:1734019268.214559:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019268.214563:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019268.214566:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019268.214572:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333126400/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019268.214579:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019268.214581:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1734019268.214582:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333126016:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019268.214583:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019268.214585:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3530 00000100:00080000:0.0:1734019268.214587:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019268.214590:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019268.214593:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019268.214599:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333126528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019268.214605:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019268.214607:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019268.214609:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019268.214612:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3530 00000100:00080000:0.0:1734019268.214614:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019268.214616:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019268.214620:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019268.214626:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333126656/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019268.214632:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019268.214635:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019268.214636:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1734019268.214637:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019268.214639:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3530 00000100:00080000:0.0:1734019268.214641:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019268.214643:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019268.214647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019268.214652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333126784/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019268.214655:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019268/real 1734019268] req@ffff8800a705e300 x1818247333126016/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019268.214670:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333126016/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019268.214679:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019268.214681:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019268.214684:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333126016:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019268.214690:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333126144:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019268.214706:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019268.214711:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019268/real 1734019268] req@ffff8800a705c000 x1818247333126144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019268.214720:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333126144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019268.214726:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019268.214728:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019268.214731:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333126144:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019268.214735:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333126272:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019268.214745:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019268.214749:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019268/real 1734019268] req@ffff8800a705e680 x1818247333126272/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019268.214757:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333126272/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019268.214762:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019268.214764:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019268.214766:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333126272:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019268.214785:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333126400:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019268.214795:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019268.214800:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019268/real 1734019268] req@ffff8800a705c700 x1818247333126400/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019268.214813:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333126400/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019268.214819:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019268.214821:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019268.214823:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333126400:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019268.214827:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333126528:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019268.214836:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019268.214840:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019268/real 1734019268] req@ffff8800a705d180 x1818247333126528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019268.214847:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333126528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019268.214853:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019268.214854:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019268.214856:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333126528:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019268.214861:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333126656:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019268.214869:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019268.214874:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019268/real 1734019268] req@ffff8800a705c380 x1818247333126656/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019268.214881:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333126656/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019268.214887:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019268.214889:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019268.214891:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333126656:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019268.214896:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333126784:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019268.214905:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019268.214909:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019268/real 1734019268] req@ffff8800a705d880 x1818247333126784/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019268.214916:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333126784/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019268.214922:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019268.214923:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019268.214925:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333126784:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019268.238500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019269.240493:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019270.242494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019271.244475:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019272.246524:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019273.222459:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019273.222469:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019273.222474:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019273.222484:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3535 00000100:00080000:2.0:1734019273.222494:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019273.222502:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734019273.222539:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247333126912/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019273.222578:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019273.222582:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019273.222584:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019273.222588:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3535 00000100:00080000:2.0:1734019273.222592:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019273.222596:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019273.222607:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019273.222619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247333127040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019273.222621:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333126912:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734019273.222630:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019273.222633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019273.222634:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019273.222638:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3535 00000100:00080000:2.0:1734019273.222640:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019273.222643:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019273.222648:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734019273.222656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247333127168/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019273.222664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019273.222667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019273.222669:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019273.222672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3535 00000100:00080000:2.0:1734019273.222677:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019273.222680:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019273.222685:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734019273.222688:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019273.222694:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333127296/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019273.222708:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019273/real 1734019273] req@ffff8800a73caa00 x1818247333126912/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019273.222709:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019273.222712:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019273.222714:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019273.222718:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3535 00000100:00080000:2.0:1734019273.222720:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019273.222724:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019273.222727:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73caa00 x1818247333126912/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019273.222728:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019273.222735:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333127424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019273.222739:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019273.222742:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:2.0:1734019273.222743:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019273.222746:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1734019273.222747:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333126912:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734019273.222748:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019273.222752:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3535 00000100:00080000:2.0:1734019273.222754:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019273.222757:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019273.222762:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019273.222769:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333127040:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019273.222769:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333127552/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019273.222777:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019273.222780:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019273.222782:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1734019273.222786:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:2.0:1734019273.222786:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3535 00000100:00080000:2.0:1734019273.222789:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019273.222792:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019273.222794:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019273/real 1734019273] req@ffff8800a73c9f80 x1818247333127040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019273.222797:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019273.222804:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333127680/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019273.222806:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9f80 x1818247333127040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019273.222815:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019273.222817:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019273.222821:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333127040:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019273.222826:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333127168:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019273.222840:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019273.222846:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019273/real 1734019273] req@ffff8800a73ca680 x1818247333127168/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019273.222856:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ca680 x1818247333127168/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019273.222863:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019273.222866:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019273.222869:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333127168:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019273.222875:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333127296:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019273.222887:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019273.222893:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019273/real 1734019273] req@ffff8800a73cad80 x1818247333127296/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019273.222903:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333127296/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019273.222910:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019273.222912:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019273.222915:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333127296:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019273.222921:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333127424:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019273.222933:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019273.222939:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019273/real 1734019273] req@ffff8800a73cb100 x1818247333127424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019273.222949:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333127424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019273.222957:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019273.222959:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019273.222962:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333127424:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019273.222976:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333127552:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019273.222988:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019273.222994:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019273/real 1734019273] req@ffff8800a73c9880 x1818247333127552/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019273.223004:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333127552/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019273.223010:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019273.223012:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019273.223015:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333127552:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019273.223021:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333127680:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019273.223033:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019273.223039:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019273/real 1734019273] req@ffff8800a73c8e00 x1818247333127680/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019273.223049:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333127680/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019273.223057:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019273.223059:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019273.223062:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333127680:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019273.248507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019274.250523:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019275.252529:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019276.254505:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019277.256499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019278.230515:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019278.230525:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019278.230529:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019278.230540:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3540 00000100:00080000:0.0:1734019278.230550:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019278.230558:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019278.230593:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333127808/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019278.230631:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019278.230635:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019278.230637:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019278.230642:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3540 00000100:00080000:0.0:1734019278.230645:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019278.230648:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019278.230658:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019278.230669:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333127936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019278.230685:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019278.230689:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019278.230690:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019278.230694:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3540 00000100:00080000:0.0:1734019278.230696:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019278.230700:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019278.230705:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019278.230713:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333128064/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019278.230721:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019278.230724:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019278.230726:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019278.230726:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333127808:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019278.230729:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3540 00000100:00080000:0.0:1734019278.230732:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019278.230735:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019278.230739:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019278.230747:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333128192/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019278.230754:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019278.230758:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019278.230759:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019278.230763:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3540 00000100:00080000:0.0:1734019278.230766:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019278.230769:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019278.230773:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019278.230780:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333128320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019278.230788:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019278.230791:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019278.230793:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1734019278.230794:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019278.230796:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3540 00000100:00080000:0.0:1734019278.230799:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019278.230802:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019278.230806:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019278.230814:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333128448/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019278.230816:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019278/real 1734019278] req@ffff8800a705f800 x1818247333127808/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019278.230822:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019278.230825:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019278.230826:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019278.230830:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3540 00000100:00080000:0.0:1734019278.230832:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019278.230836:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:2.0:1734019278.230836:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333127808/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734019278.230840:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1734019278.230847:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1734019278.230849:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333128576/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019278.230850:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019278.230855:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333127808:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019278.230862:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333127936:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019278.230879:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019278.230886:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019278/real 1734019278] req@ffff8800a705d500 x1818247333127936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019278.230897:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333127936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019278.230905:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019278.230908:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019278.230912:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333127936:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019278.230936:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333128064:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019278.230950:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019278.230956:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019278/real 1734019278] req@ffff8800a705ed80 x1818247333128064/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019278.230966:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333128064/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019278.230974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019278.230976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019278.230979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333128064:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019278.230984:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333128192:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019278.230996:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019278.231001:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019278/real 1734019278] req@ffff8800a705ea00 x1818247333128192/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019278.231011:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333128192/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019278.231019:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019278.231021:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019278.231023:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333128192:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019278.231028:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333128320:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019278.231040:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019278.231047:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019278/real 1734019278] req@ffff8800a705fb80 x1818247333128320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019278.231057:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333128320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019278.231065:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019278.231067:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019278.231070:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333128320:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019278.231074:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333128448:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019278.231086:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019278.231092:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019278/real 1734019278] req@ffff8800a705dc00 x1818247333128448/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019278.231102:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333128448/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019278.231109:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019278.231111:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019278.231114:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333128448:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019278.231119:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333128576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019278.231130:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019278.231136:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019278/real 1734019278] req@ffff8800a705f100 x1818247333128576/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019278.231146:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333128576/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019278.231161:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019278.231163:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019278.231166:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333128576:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019278.258477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019279.260491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019280.262508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019281.264511:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019282.266454:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019283.238345:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019283.238355:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019283.238359:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019283.238369:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3545 00000100:00080000:0.0:1734019283.238379:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019283.238386:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019283.238422:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333128704/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019283.238459:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019283.238465:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019283.238468:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019283.238475:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3545 00000100:00080000:0.0:1734019283.238480:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019283.238486:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019283.238499:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019283.238517:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333128832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019283.238533:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019283.238537:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1734019283.238537:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333128704:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019283.238540:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019283.238545:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3545 00000100:00080000:0.0:1734019283.238549:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019283.238554:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019283.238562:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019283.238576:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333128960/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019283.238588:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019283.238592:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019283.238595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019283.238599:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3545 00000100:00080000:0.0:1734019283.238603:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734019283.238606:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019283.238608:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019283.238615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019283.238626:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333129088/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019283.238630:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019283/real 1734019283] req@ffff8800a705df80 x1818247333128704/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019283.238640:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019283.238645:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019283.238647:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019283.238652:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3545 00000100:00100000:2.0:1734019283.238655:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333128704/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019283.238657:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019283.238662:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019283.238669:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1734019283.238669:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019283.238674:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019283.238680:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333128704:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019283.238681:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333129216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019283.238694:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019283.238698:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019283.238701:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019283.238707:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3545 00000100:00100000:2.0:1734019283.238707:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333128832:192.168.201.152@tcp:38:kworker.0 00000100:00080000:0.0:1734019283.238711:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019283.238716:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019283.238723:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019283.238733:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019283.238735:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333129344/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019283.238745:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019283/real 1734019283] req@ffff8800a705ca80 x1818247333128832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019283.238749:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019283.238754:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019283.238757:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019283.238762:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333128832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019283.238763:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3545 00000100:00080000:0.0:1734019283.238767:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019283.238772:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:2.0:1734019283.238775:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019283.238778:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000008:00100000:0.0:1734019283.238779:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019283.238784:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333128832:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019283.238790:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333129472/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019283.238794:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333128960:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019283.238817:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019283.238828:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019283/real 1734019283] req@ffff8800a705f100 x1818247333128960/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019283.238845:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333128960/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019283.238857:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019283.238861:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019283.238866:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333128960:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019283.238875:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333129088:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019283.238899:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019283.238908:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019283/real 1734019283] req@ffff8800a705dc00 x1818247333129088/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019283.238919:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333129088/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019283.238927:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019283.238929:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019283.238932:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333129088:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019283.238937:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333129216:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019283.238949:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019283.238955:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019283/real 1734019283] req@ffff8800a705fb80 x1818247333129216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019283.238966:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333129216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019283.238974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019283.238976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019283.238979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333129216:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019283.238996:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333129344:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019283.239009:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019283.239015:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019283/real 1734019283] req@ffff8800a705ea00 x1818247333129344/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019283.239024:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333129344/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019283.239032:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019283.239034:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019283.239037:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333129344:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019283.239042:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333129472:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019283.239053:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019283.239059:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019283/real 1734019283] req@ffff8800a705ed80 x1818247333129472/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019283.239068:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333129472/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019283.239076:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019283.239088:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019283.239092:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333129472:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019283.268466:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019284.270480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019285.272477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019286.274474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019287.276475:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019288.246371:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019288.246381:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019288.246385:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019288.246396:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3550 00000100:00080000:0.0:1734019288.246406:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019288.246414:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019288.246450:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333129600/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019288.246493:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019288.246497:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019288.246499:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019288.246503:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3550 00000100:00080000:0.0:1734019288.246507:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019288.246510:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019288.246521:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019288.246531:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333129728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019288.246541:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019288.246544:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019288.246546:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019288.246549:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3550 00000100:00080000:0.0:1734019288.246552:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019288.246555:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019288.246560:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019288.246568:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333129856/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019288.246576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019288.246579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019288.246581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019288.246584:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3550 00000100:00080000:0.0:1734019288.246586:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019288.246590:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019288.246594:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019288.246602:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333129984/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019288.246611:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019288.246614:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1734019288.246615:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333129600:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019288.246616:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019288.246619:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3550 00000100:00080000:0.0:1734019288.246622:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019288.246625:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019288.246629:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019288.246636:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333130112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019288.246644:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019288.246647:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019288.246649:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019288.246653:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3550 00000100:00080000:0.0:1734019288.246655:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019288.246658:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019288.246663:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019288.246670:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333130240/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019288.246678:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019288.246681:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1734019288.246682:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019288.246683:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019288.246687:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3550 00000100:00080000:0.0:1734019288.246690:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019288.246693:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019288.246697:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019288.246704:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019288/real 1734019288] req@ffff8800a705d500 x1818247333129600/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019288.246705:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333130368/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019288.246724:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333129600/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019288.246735:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019288.246738:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019288.246743:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333129600:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019288.246750:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333129728:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019288.246768:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019288.246774:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019288/real 1734019288] req@ffff8800a705f800 x1818247333129728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019288.246786:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333129728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019288.246794:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019288.246797:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019288.246800:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333129728:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019288.246806:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333129856:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019288.246822:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019288.246828:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019288/real 1734019288] req@ffff8800a705d880 x1818247333129856/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019288.246839:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333129856/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019288.246847:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019288.246849:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019288.246852:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333129856:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019288.246875:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333129984:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019288.246887:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019288.246893:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019288/real 1734019288] req@ffff8800a705c380 x1818247333129984/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019288.246903:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333129984/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019288.246910:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019288.246913:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019288.246916:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333129984:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019288.246920:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333130112:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019288.246932:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019288.246938:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019288/real 1734019288] req@ffff8800a705d180 x1818247333130112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019288.246974:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333130112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019288.246982:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019288.246984:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019288.246987:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333130112:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019288.246992:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333130240:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019288.247003:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019288.247008:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019288/real 1734019288] req@ffff8800a705c700 x1818247333130240/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019288.247018:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333130240/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019288.247025:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019288.247027:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019288.247030:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333130240:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019288.247035:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333130368:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019288.247047:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019288.247053:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019288/real 1734019288] req@ffff8800a705e680 x1818247333130368/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019288.247063:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333130368/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019288.247070:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019288.247074:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019288.247077:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333130368:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019288.278451:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019289.280491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019290.282494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019291.284492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019292.286509:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019293.254363:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019293.254373:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019293.254378:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019293.254389:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3555 00000100:00080000:0.0:1734019293.254399:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019293.254407:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019293.254445:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333130496/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019293.254481:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019293.254485:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019293.254487:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019293.254492:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3555 00000100:00080000:0.0:1734019293.254496:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019293.254499:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019293.254509:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019293.254521:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333130624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019293.254532:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019293.254535:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019293.254537:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019293.254540:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3555 00000100:00080000:0.0:1734019293.254542:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019293.254546:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019293.254550:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019293.254559:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333130752/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019293.254567:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019293.254570:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019293.254572:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019293.254575:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3555 00000100:00080000:0.0:1734019293.254577:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019293.254581:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019293.254585:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019293.254592:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333130880/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019293.254600:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333130496:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019293.254601:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019293.254604:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019293.254606:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019293.254610:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3555 00000100:00080000:0.0:1734019293.254613:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019293.254616:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019293.254621:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019293.254629:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333131008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019293.254637:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019293.254640:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019293.254641:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019293.254645:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3555 00000100:00080000:0.0:1734019293.254648:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019293.254651:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019293.254655:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019293.254662:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333131136/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019293.254670:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019293.254673:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019293.254674:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019293.254677:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3555 00000100:00080000:0.0:1734019293.254680:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019293.254683:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019293.254687:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019293.254694:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333130624:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019293.254695:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333131264/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019293.254715:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333130752:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019293.254746:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333130880:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019293.254760:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333131008:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019293.254774:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333131136:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019293.254788:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333131264:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019293.288503:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019294.290497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019295.292485:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1734019296.260511:0:1955:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:3.0:1734019296.260558:0:1955:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1734019296.260580:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734019296.260615:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734019296.260623:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734019296.260631:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734019296.260638:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734019296.260645:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:3.0:1734019296.260652:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:2.0:1734019296.260723:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019293/real 1734019296] req@ffff8800a705c700 x1818247333130880/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019296.260753:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333130880/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019296.260765:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019296.260769:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019296.260774:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333130880:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019296.260782:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019293/real 1734019296] req@ffff8800a705d180 x1818247333131008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019296.260794:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333131008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019296.260803:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019296.260806:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019296.260810:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333131008:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019296.260816:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019293/real 1734019296] req@ffff8800a705c380 x1818247333131136/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019296.260827:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333131136/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019296.260834:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019296.260836:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019296.260840:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333131136:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019296.260845:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019293/real 1734019296] req@ffff8800a705d880 x1818247333131264/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019296.260856:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333131264/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019296.260863:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019296.260865:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019296.260868:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333131264:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019296.260874:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019293/real 1734019296] req@ffff8800a705c000 x1818247333130496/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019296.260885:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333130496/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019296.260900:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019296.260903:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019296.260906:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333130496:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019296.260912:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019293/real 1734019296] req@ffff8800a705e300 x1818247333130624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019296.260922:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333130624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019296.260929:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019296.260932:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019296.260935:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333130624:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019296.260941:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019293/real 1734019296] req@ffff8800a705e680 x1818247333130752/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019296.261008:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333130752/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019296.261016:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019296.261018:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019296.261021:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333130752:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019296.294456:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019297.296445:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019298.262340:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019298.262355:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019298.262361:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019298.262375:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3560 00000100:00080000:0.0:1734019298.262388:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019298.262397:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019298.262448:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333131392/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019298.262488:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019298.262493:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019298.262495:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019298.262501:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3560 00000100:00080000:0.0:1734019298.262504:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019298.262508:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019298.262520:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019298.262531:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333131520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019298.262542:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019298.262545:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019298.262547:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019298.262551:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3560 00000100:00080000:0.0:1734019298.262554:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019298.262558:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019298.262563:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019298.262573:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333131648/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019298.262582:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019298.262585:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019298.262587:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019298.262591:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3560 00000100:00080000:0.0:1734019298.262593:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019298.262597:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019298.262601:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019298.262609:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333131776/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019298.262619:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019298.262623:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019298.262624:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019298.262628:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3560 00000100:00080000:0.0:1734019298.262630:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019298.262634:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019298.262639:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019298.262643:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333131392:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019298.262647:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333131904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019298.262656:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019298.262660:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019298.262662:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019298.262667:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3560 00000100:00080000:0.0:1734019298.262670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019298.262673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019298.262678:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019298.262686:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333132032/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019298.262694:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019298.262697:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019298.262699:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019298.262703:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3560 00000100:00080000:0.0:1734019298.262706:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019298.262709:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019298.262714:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019298.262721:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019298.262724:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333132160/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019298.262747:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019298/real 1734019298] req@ffff8800a705f800 x1818247333131392/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019298.262775:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333131392/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019298.262787:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019298.262790:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019298.262795:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333131392:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019298.262803:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333131520:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019298.262821:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019298.262827:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019298/real 1734019298] req@ffff8800a705d500 x1818247333131520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019298.262839:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333131520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019298.262847:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019298.262849:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019298.262853:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333131520:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019298.262859:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333131648:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019298.262873:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019298.262879:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019298/real 1734019298] req@ffff8800a705ed80 x1818247333131648/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019298.262889:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333131648/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019298.262896:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019298.262900:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019298.262903:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333131648:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019298.262973:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333131776:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019298.262990:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019298.262997:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019298/real 1734019298] req@ffff8800a705ea00 x1818247333131776/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019298.263008:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333131776/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019298.263016:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019298.263018:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019298.263021:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333131776:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019298.263027:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333131904:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019298.263039:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019298.263045:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019298/real 1734019298] req@ffff8800a705fb80 x1818247333131904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019298.263055:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333131904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019298.263063:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019298.263066:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019298.263069:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333131904:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019298.263075:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333132032:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019298.263090:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019298.263096:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019298/real 1734019298] req@ffff8800a705dc00 x1818247333132032/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019298.263106:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333132032/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019298.263113:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019298.263115:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019298.263118:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333132032:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019298.263124:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333132160:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019298.263136:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019298.263142:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019298/real 1734019298] req@ffff8800a705f100 x1818247333132160/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019298.263152:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333132160/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019298.263159:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019298.263162:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019298.263165:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333132160:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019298.298513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019299.300466:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019300.302499:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019301.304477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019302.306510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019303.270349:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019303.270359:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019303.270363:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019303.270375:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3565 00000100:00080000:0.0:1734019303.270385:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019303.270393:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019303.270430:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333132288/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019303.270473:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019303.270477:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019303.270479:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019303.270484:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3565 00000100:00080000:0.0:1734019303.270488:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019303.270491:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019303.270501:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019303.270512:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333132416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019303.270522:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019303.270525:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019303.270527:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019303.270531:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3565 00000100:00080000:0.0:1734019303.270533:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019303.270536:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019303.270541:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019303.270551:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333132544/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019303.270559:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019303.270562:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019303.270564:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019303.270567:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3565 00000100:00080000:0.0:1734019303.270570:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019303.270574:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019303.270578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019303.270586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333132672/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019303.270595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734019303.270597:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333132288:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019303.270598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019303.270600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019303.270603:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3565 00000100:00080000:0.0:1734019303.270606:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019303.270609:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019303.270613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019303.270621:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333132800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019303.270629:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019303.270632:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019303.270633:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019303.270637:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3565 00000100:00080000:0.0:1734019303.270639:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019303.270642:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019303.270647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019303.270654:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333132928/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019303.270662:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1734019303.270665:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019303.270666:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019303.270668:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019303.270671:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3565 00000100:00080000:0.0:1734019303.270674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019303.270677:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019303.270681:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019303.270686:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019303/real 1734019303] req@ffff8800a705ca80 x1818247333132288/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019303.270689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333133056/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019303.270713:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333132288/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019303.270723:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019303.270727:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019303.270731:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333132288:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019303.270739:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333132416:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019303.270756:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019303.270763:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019303/real 1734019303] req@ffff8800a705df80 x1818247333132416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019303.270774:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333132416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019303.270781:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019303.270784:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019303.270787:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333132416:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019303.270793:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333132544:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019303.270806:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019303.270812:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019303/real 1734019303] req@ffff8800a705f100 x1818247333132544/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019303.270822:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333132544/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019303.270865:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019303.270868:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019303.270872:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333132544:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019303.270898:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333132672:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019303.270915:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019303.270923:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019303/real 1734019303] req@ffff8800a705dc00 x1818247333132672/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019303.270934:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333132672/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019303.270942:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019303.270944:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019303.270947:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333132672:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019303.270952:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333132800:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019303.270964:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019303.270969:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019303/real 1734019303] req@ffff8800a705fb80 x1818247333132800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019303.270979:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333132800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019303.270987:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019303.270989:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019303.270992:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333132800:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019303.270997:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333132928:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019303.271012:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019303.271018:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019303/real 1734019303] req@ffff8800a705ea00 x1818247333132928/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019303.271029:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333132928/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019303.271036:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019303.271039:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019303.271042:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333132928:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019303.271046:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333133056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019303.271058:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019303.271064:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019303/real 1734019303] req@ffff8800a705ed80 x1818247333133056/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019303.271073:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333133056/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019303.271080:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019303.271082:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019303.271085:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333133056:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019303.308476:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019304.310465:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019305.312513:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019306.314461:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019307.316495:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019308.278382:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019308.278392:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019308.278397:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019308.278408:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3570 00000100:00080000:0.0:1734019308.278419:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019308.278427:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019308.278465:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333133184/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019308.278502:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019308.278507:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019308.278509:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019308.278513:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3570 00000100:00080000:0.0:1734019308.278516:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019308.278520:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019308.278531:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019308.278542:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333133312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019308.278553:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019308.278557:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019308.278558:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019308.278562:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3570 00000100:00080000:0.0:1734019308.278565:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019308.278568:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019308.278573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019308.278581:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333133440/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019308.278590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019308.278593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019308.278595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019308.278598:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3570 00000100:00080000:0.0:1734019308.278600:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019308.278604:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019308.278608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019308.278615:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333133568/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019308.278622:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333133184:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019308.278624:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019308.278627:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019308.278629:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019308.278633:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3570 00000100:00080000:0.0:1734019308.278636:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019308.278639:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019308.278643:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019308.278651:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333133696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019308.278659:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019308.278662:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019308.278664:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019308.278667:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3570 00000100:00080000:0.0:1734019308.278670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019308.278673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019308.278677:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019308.278686:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333133824/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734019308.278690:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019308.278694:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019308.278697:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019308.278698:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019308.278702:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3570 00000100:00080000:0.0:1734019308.278705:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019308.278708:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:2.0:1734019308.278711:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019308/real 1734019308] req@ffff8800a705d500 x1818247333133184/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:0.0:1734019308.278712:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019308.278720:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333133952/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019308.278738:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333133184/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019308.278790:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019308.278793:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019308.278799:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333133184:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019308.278807:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333133312:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019308.278835:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019308.278844:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019308/real 1734019308] req@ffff8800a705f800 x1818247333133312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019308.278855:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333133312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019308.278863:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019308.278866:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019308.278869:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333133312:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019308.278875:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333133440:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019308.278888:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019308.278894:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019308/real 1734019308] req@ffff8800a705e680 x1818247333133440/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019308.278904:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333133440/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019308.278912:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019308.278915:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019308.278918:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333133440:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019308.278942:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333133568:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019308.278955:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019308.278961:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019308/real 1734019308] req@ffff8800a705e300 x1818247333133568/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019308.278971:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333133568/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019308.278979:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019308.278981:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019308.278984:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333133568:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019308.278989:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333133696:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019308.279000:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019308.279006:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019308/real 1734019308] req@ffff8800a705c000 x1818247333133696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019308.279015:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333133696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019308.279023:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019308.279025:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019308.279028:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333133696:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019308.279033:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333133824:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019308.279047:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019308.279053:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019308/real 1734019308] req@ffff8800a705d880 x1818247333133824/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019308.279063:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333133824/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019308.279070:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019308.279072:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019308.279075:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333133824:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019308.279080:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333133952:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019308.279092:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019308.279097:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019308/real 1734019308] req@ffff8800a705c380 x1818247333133952/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019308.279107:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333133952/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019308.279114:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019308.279116:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019308.279119:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333133952:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019308.318474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019309.320487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019310.322479:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019311.324479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019312.326410:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019313.286358:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019313.286368:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019313.286372:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019313.286383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3575 00000100:00080000:0.0:1734019313.286393:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019313.286401:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019313.286441:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333134080/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019313.286488:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019313.286494:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019313.286498:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019313.286505:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3575 00000100:00080000:0.0:1734019313.286510:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019313.286517:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019313.286530:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019313.286550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333134208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019313.286566:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019313.286571:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019313.286574:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019313.286580:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3575 00000100:00080000:0.0:1734019313.286585:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019313.286589:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019313.286596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734019313.286601:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333134080:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019313.286609:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333134336/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019313.286619:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019313.286623:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019313.286625:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019313.286628:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3575 00000100:00080000:0.0:1734019313.286631:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019313.286634:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019313.286639:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019313.286647:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333134464/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019313.286655:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019313.286688:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019313.286691:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019313.286695:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3575 00000100:00080000:0.0:1734019313.286698:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019313.286702:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019313.286706:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1734019313.286710:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019313.286716:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333134592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019313.286725:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019313.286728:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019313.286730:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019313.286734:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019313/real 1734019313] req@ffff8800a705d180 x1818247333134080/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019313.286735:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3575 00000100:00080000:0.0:1734019313.286737:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019313.286741:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019313.286745:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019313.286752:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333134720/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019313.286760:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734019313.286762:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333134080/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019313.286763:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019313.286765:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019313.286768:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3575 00000100:00080000:0.0:1734019313.286771:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019313.286773:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019313.286774:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:2.0:1734019313.286776:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000008:00100000:0.0:1734019313.286778:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019313.286781:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333134080:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019313.286786:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333134848/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019313.286789:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333134208:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019313.286808:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019313.286816:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019313/real 1734019313] req@ffff8800a705c700 x1818247333134208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019313.286827:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333134208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019313.286835:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019313.286838:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019313.286841:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333134208:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019313.286863:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333134336:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019313.286877:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019313.286883:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019313/real 1734019313] req@ffff8800a705c380 x1818247333134336/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019313.286893:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333134336/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019313.286901:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019313.286905:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019313.286908:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333134336:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019313.286913:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333134464:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019313.286925:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019313.286931:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019313/real 1734019313] req@ffff8800a705d880 x1818247333134464/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019313.286941:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333134464/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019313.286949:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019313.286951:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019313.286954:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333134464:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019313.286959:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333134592:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019313.286970:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019313.286976:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019313/real 1734019313] req@ffff8800a705c000 x1818247333134592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019313.286986:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333134592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019313.286993:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019313.286995:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019313.286998:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333134592:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019313.287003:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333134720:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019313.287023:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019313.287029:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019313/real 1734019313] req@ffff8800a705e300 x1818247333134720/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019313.287040:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333134720/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019313.287047:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019313.287049:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019313.287052:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333134720:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019313.287057:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333134848:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019313.287068:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019313.287073:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019313/real 1734019313] req@ffff8800a705e680 x1818247333134848/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019313.287083:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333134848/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019313.287092:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019313.287094:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019313.287097:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333134848:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019313.328455:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019314.330468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019315.332522:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019316.334498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019317.336477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019318.294349:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019318.294362:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019318.294367:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019318.294382:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3580 00000100:00080000:0.0:1734019318.294394:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019318.294404:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019318.294452:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333134976/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019318.294500:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019318.294505:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019318.294506:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019318.294511:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3580 00000100:00080000:0.0:1734019318.294515:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019318.294519:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019318.294531:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019318.294543:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333135104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019318.294552:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019318.294556:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019318.294558:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019318.294562:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3580 00000100:00080000:0.0:1734019318.294565:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019318.294568:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019318.294618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019318.294629:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333135232/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019318.294638:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019318.294642:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019318.294644:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019318.294648:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3580 00000100:00080000:0.0:1734019318.294651:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019318.294655:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019318.294660:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019318.294670:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333135360/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019318.294676:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333134976:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019318.294680:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019318.294684:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019318.294685:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019318.294689:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3580 00000100:00080000:0.0:1734019318.294692:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019318.294696:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019318.294701:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019318.294710:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333135488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019318.294718:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019318.294721:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019318.294722:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019318.294726:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3580 00000100:00080000:0.0:1734019318.294729:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019318.294733:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019318.294737:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019318.294745:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333135616/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019318.294753:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1734019318.294755:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019318.294757:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019318.294759:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019318.294763:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3580 00000100:00080000:0.0:1734019318.294767:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019318.294770:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019318.294774:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019318.294780:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019318/real 1734019318] req@ffff8800a705f800 x1818247333134976/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019318.294785:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333135744/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019318.294808:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333134976/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019318.294821:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019318.294824:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019318.294829:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333134976:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019318.294837:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333135104:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019318.294856:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019318.294862:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019318/real 1734019318] req@ffff8800a705d500 x1818247333135104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019318.294874:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333135104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019318.294882:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019318.294885:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019318.294888:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333135104:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019318.294894:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333135232:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019318.294908:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019318.294914:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019318/real 1734019318] req@ffff8800a705ed80 x1818247333135232/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019318.294924:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333135232/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019318.294931:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019318.294934:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019318.294938:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333135232:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019318.294965:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333135360:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019318.294978:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019318.294985:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019318/real 1734019318] req@ffff8800a705ea00 x1818247333135360/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019318.294995:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333135360/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019318.295003:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019318.295005:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019318.295008:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333135360:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019318.295014:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333135488:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019318.295026:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019318.295032:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019318/real 1734019318] req@ffff8800a705fb80 x1818247333135488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019318.295041:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333135488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019318.295049:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019318.295052:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019318.295055:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333135488:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019318.295061:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333135616:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019318.295076:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019318.295082:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019318/real 1734019318] req@ffff8800a705dc00 x1818247333135616/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019318.295093:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333135616/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019318.295100:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019318.295102:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019318.295105:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333135616:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019318.295111:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333135744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019318.295123:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019318.295128:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019318/real 1734019318] req@ffff8800a705f100 x1818247333135744/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019318.295138:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333135744/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019318.295145:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019318.295148:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019318.295150:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333135744:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019318.338474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019319.340466:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019320.342466:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019321.344449:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019322.346474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019323.302344:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019323.302355:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019323.302360:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019323.302375:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3585 00000100:00080000:0.0:1734019323.302387:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019323.302397:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019323.302446:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705df80 x1818247333135872/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019323.302484:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019323.302519:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019323.302522:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019323.302527:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3585 00000100:00080000:0.0:1734019323.302531:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019323.302536:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019323.302549:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019323.302563:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ca80 x1818247333136000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019323.302575:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019323.302579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019323.302581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019323.302584:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3585 00000100:00080000:0.0:1734019323.302587:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019323.302591:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019323.302596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734019323.302605:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333135872:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019323.302607:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f100 x1818247333136128/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019323.302618:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019323.302622:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019323.302624:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019323.302628:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3585 00000100:00080000:0.0:1734019323.302630:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019323.302634:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019323.302639:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019323.302647:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705dc00 x1818247333136256/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019323.302655:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019323.302659:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019323.302660:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019323.302664:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3585 00000100:00080000:0.0:1734019323.302668:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019323.302672:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019323.302677:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1734019323.302681:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019323.302686:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705fb80 x1818247333136384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019323.302694:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019323.302697:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019323.302698:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019323.302702:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3585 00000100:00080000:0.0:1734019323.302705:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734019323.302706:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019323/real 1734019323] req@ffff8800a705df80 x1818247333135872/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019323.302709:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019323.302714:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019323.302722:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ea00 x1818247333136512/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019323.302730:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019323.302734:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1734019323.302734:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705df80 x1818247333135872/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019323.302735:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019323.302739:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3585 00000100:00080000:0.0:1734019323.302741:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019323.302745:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:2.0:1734019323.302746:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1734019323.302749:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1734019323.302750:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019323.302755:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333135872:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019323.302758:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705ed80 x1818247333136640/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019323.302763:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333136000:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019323.302782:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019323.302789:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019323/real 1734019323] req@ffff8800a705ca80 x1818247333136000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019323.302800:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ca80 x1818247333136000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019323.302808:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019323.302811:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019323.302814:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333136000:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019323.302840:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333136128:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019323.302854:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019323.302860:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019323/real 1734019323] req@ffff8800a705f100 x1818247333136128/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019323.302870:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f100 x1818247333136128/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019323.302878:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019323.302882:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019323.302885:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333136128:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019323.302891:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333136256:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019323.302903:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019323.302909:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019323/real 1734019323] req@ffff8800a705dc00 x1818247333136256/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019323.302919:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705dc00 x1818247333136256/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019323.302927:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019323.302929:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019323.302932:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333136256:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019323.302938:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333136384:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019323.302951:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019323.302956:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019323/real 1734019323] req@ffff8800a705fb80 x1818247333136384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019323.302966:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705fb80 x1818247333136384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019323.302974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019323.302976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019323.302979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333136384:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019323.302985:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333136512:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019323.303001:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019323.303007:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019323/real 1734019323] req@ffff8800a705ea00 x1818247333136512/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019323.303017:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ea00 x1818247333136512/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019323.303024:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019323.303027:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019323.303030:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333136512:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019323.303035:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333136640:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019323.303047:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019323.303052:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019323/real 1734019323] req@ffff8800a705ed80 x1818247333136640/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019323.303062:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705ed80 x1818247333136640/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019323.303069:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019323.303071:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019323.303074:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333136640:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019323.348447:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019324.350462:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019325.352506:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019326.354489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019327.356482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019328.310466:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019328.310478:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019328.310483:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019328.310494:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3590 00000100:00080000:0.0:1734019328.310506:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019328.310514:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019328.310552:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d500 x1818247333136768/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019328.310590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019328.310594:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019328.310597:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019328.310601:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3590 00000100:00080000:0.0:1734019328.310604:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019328.310608:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019328.310618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019328.310628:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705f800 x1818247333136896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019328.310639:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019328.310642:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019328.310643:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019328.310647:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3590 00000100:00080000:0.0:1734019328.310650:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019328.310653:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019328.310659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019328.310667:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333137024/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019328.310676:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019328.310678:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019328.310680:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019328.310683:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3590 00000100:00080000:0.0:1734019328.310686:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019328.310689:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019328.310694:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019328.310701:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333137152/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019328.310710:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019328.310713:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019328.310715:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019328.310718:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3590 00000100:00100000:2.0:1734019328.310718:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333136768:192.168.201.152@tcp:250:kworker.0 00000100:00080000:0.0:1734019328.310721:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019328.310724:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019328.310728:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019328.310736:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333137280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019328.310744:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019328.310746:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019328.310749:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019328.310752:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3590 00000100:00080000:0.0:1734019328.310755:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019328.310758:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019328.310762:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019328.310771:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333137408/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019328.310779:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019328.310782:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019328.310784:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1734019328.310786:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:0.0:1734019328.310787:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3590 00000100:00080000:0.0:1734019328.310790:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019328.310793:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019328.310797:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019328.310804:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333137536/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019328.310809:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019328/real 1734019328] req@ffff8800a705d500 x1818247333136768/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019328.310836:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d500 x1818247333136768/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019328.310847:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019328.310850:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019328.310855:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333136768:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019328.310862:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333136896:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019328.310878:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019328.310884:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019328/real 1734019328] req@ffff8800a705f800 x1818247333136896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019328.310895:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705f800 x1818247333136896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019328.310903:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019328.310906:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019328.310909:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333136896:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019328.310915:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333137024:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019328.310928:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019328.310934:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019328/real 1734019328] req@ffff8800a705e680 x1818247333137024/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019328.310944:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333137024/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019328.310951:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019328.310954:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019328.310957:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333137024:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019328.310982:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333137152:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019328.310994:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019328.311001:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019328/real 1734019328] req@ffff8800a705e300 x1818247333137152/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019328.311011:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333137152/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019328.311019:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019328.311021:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019328.311024:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333137152:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019328.311029:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333137280:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019328.311040:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019328.311046:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019328/real 1734019328] req@ffff8800a705c000 x1818247333137280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019328.311055:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333137280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019328.311063:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019328.311065:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019328.311068:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333137280:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019328.311073:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333137408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019328.311088:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019328.311095:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019328/real 1734019328] req@ffff8800a705d880 x1818247333137408/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019328.311105:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333137408/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019328.311112:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019328.311114:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019328.311117:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333137408:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019328.311122:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333137536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019328.311133:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019328.311139:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019328/real 1734019328] req@ffff8800a705c380 x1818247333137536/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019328.311149:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333137536/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019328.311156:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019328.311158:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019328.311161:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333137536:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019328.358521:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019329.360545:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019330.362532:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019331.364525:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019332.366517:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1734019333.318386:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019333.318398:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1734019333.318402:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019333.318413:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3595 00000100:00080000:0.0:1734019333.318424:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019333.318432:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:0.0:1734019333.318468:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c700 x1818247333137664/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019333.318507:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019333.318512:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019333.318514:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019333.318518:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3595 00000100:00080000:0.0:1734019333.318522:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019333.318525:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019333.318535:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019333.318546:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d180 x1818247333137792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019333.318556:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019333.318560:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019333.318561:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019333.318565:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3595 00000100:00080000:0.0:1734019333.318567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019333.318571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019333.318576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019333.318586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c380 x1818247333137920/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019333.318594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019333.318597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019333.318598:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019333.318602:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3595 00000100:00080000:0.0:1734019333.318604:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019333.318608:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019333.318613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019333.318619:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333137664:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019333.318621:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705d880 x1818247333138048/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019333.318630:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019333.318633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1734019333.318634:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019333.318638:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3595 00000100:00080000:0.0:1734019333.318641:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019333.318644:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019333.318648:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019333.318656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705c000 x1818247333138176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019333.318664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019333.318667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1734019333.318669:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019333.318672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3595 00000100:00080000:0.0:1734019333.318675:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019333.318678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019333.318682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019333.318687:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019333.318690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e300 x1818247333138304/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019333.318698:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019333.318702:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1734019333.318704:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019333.318707:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3595 00000100:00100000:2.0:1734019333.318709:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019333/real 1734019333] req@ffff8800a705c700 x1818247333137664/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019333.318710:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019333.318714:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:0.0:1734019333.318718:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019333.318725:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a705e680 x1818247333138432/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019333.318736:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c700 x1818247333137664/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019333.318746:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019333.318750:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019333.318754:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333137664:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019333.318761:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333137792:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019333.318778:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019333.318785:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019333/real 1734019333] req@ffff8800a705d180 x1818247333137792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019333.318796:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d180 x1818247333137792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019333.318804:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019333.318806:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019333.318809:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333137792:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019333.318815:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333137920:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019333.318828:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019333.318834:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019333/real 1734019333] req@ffff8800a705c380 x1818247333137920/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019333.318844:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c380 x1818247333137920/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019333.318851:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019333.318854:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019333.318857:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333137920:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019333.318881:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333138048:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019333.318894:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019333.318900:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019333/real 1734019333] req@ffff8800a705d880 x1818247333138048/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019333.318910:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705d880 x1818247333138048/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019333.318917:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019333.318920:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019333.318923:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333138048:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019333.318927:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333138176:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019333.318939:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019333.318944:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019333/real 1734019333] req@ffff8800a705c000 x1818247333138176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019333.318954:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705c000 x1818247333138176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019333.318961:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019333.318963:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019333.318966:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333138176:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019333.318971:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333138304:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019333.318986:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019333.318992:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019333/real 1734019333] req@ffff8800a705e300 x1818247333138304/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019333.319002:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e300 x1818247333138304/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019333.319009:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019333.319011:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019333.319014:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333138304:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019333.319019:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333138432:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019333.319030:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019333.319035:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019333/real 1734019333] req@ffff8800a705e680 x1818247333138432/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019333.319045:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a705e680 x1818247333138432/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019333.319052:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019333.319054:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019333.319057:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a705e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333138432:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019333.368490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019334.370468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019335.372474:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019336.374503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019337.376514:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019338.326458:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019338.326469:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019338.326475:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019338.326485:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3600 00000100:00080000:2.0:1734019338.326497:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019338.326506:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734019338.326544:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247333138560/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019338.326585:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019338.326589:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019338.326591:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019338.326596:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3600 00000100:00080000:2.0:1734019338.326599:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019338.326603:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019338.326613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019338.326624:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247333138688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019338.326635:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019338.326638:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019338.326640:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019338.326643:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3600 00000100:00080000:2.0:1734019338.326646:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019338.326649:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019338.326654:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734019338.326662:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247333138816/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019338.326671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019338.326674:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019338.326676:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019338.326680:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3600 00000100:00080000:2.0:1734019338.326683:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019338.326686:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019338.326690:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019338.326699:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333138944/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019338.326709:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019338.326712:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1734019338.326713:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333138560:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734019338.326713:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019338.326717:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3600 00000100:00080000:2.0:1734019338.326720:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019338.326723:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019338.326728:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019338.326736:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333139072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019338.326745:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019338.326748:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019338.326749:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019338.326753:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3600 00000100:00080000:2.0:1734019338.326756:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019338.326759:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019338.326764:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019338.326771:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333139200/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019338.326780:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019338.326783:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019338.326785:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019338.326788:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3600 00000100:00080000:2.0:1734019338.326793:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019338.326796:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019338.326800:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734019338.326806:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019338.326808:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333139328/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019338.326833:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019338/real 1734019338] req@ffff8800a73c9c00 x1818247333138560/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019338.326854:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247333138560/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019338.326867:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019338.326871:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019338.326876:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333138560:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019338.326884:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333138688:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019338.326904:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019338.326911:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019338/real 1734019338] req@ffff8800a73c9500 x1818247333138688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019338.326922:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247333138688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019338.326931:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019338.326935:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019338.326938:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333138688:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019338.326945:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333138816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019338.326959:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019338.326965:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019338/real 1734019338] req@ffff8800a73c8000 x1818247333138816/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019338.326975:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247333138816/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019338.326983:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019338.326986:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019338.326989:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333138816:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019338.327019:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333138944:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019338.327032:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019338.327039:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019338/real 1734019338] req@ffff8800a73c8e00 x1818247333138944/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019338.327050:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333138944/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019338.327057:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019338.327060:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019338.327063:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333138944:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019338.327069:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333139072:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019338.327083:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019338.327089:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019338/real 1734019338] req@ffff8800a73c9880 x1818247333139072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019338.327099:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333139072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019338.327108:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019338.327110:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019338.327113:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333139072:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019338.327119:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333139200:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019338.327131:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019338.327137:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019338/real 1734019338] req@ffff8800a73cb100 x1818247333139200/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019338.327147:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333139200/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019338.327154:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019338.327156:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019338.327159:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333139200:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019338.327165:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333139328:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019338.327179:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019338.327185:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019338/real 1734019338] req@ffff8800a73cad80 x1818247333139328/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019338.327196:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333139328/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019338.327204:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019338.327206:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019338.327209:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333139328:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019338.378512:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019339.380604:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019340.382474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019341.384497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019342.386467:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019343.334342:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019343.334353:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019343.334358:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019343.334369:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3605 00000100:00080000:2.0:1734019343.334380:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019343.334388:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734019343.334425:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247333139456/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019343.334460:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019343.334465:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019343.334467:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019343.334471:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3605 00000100:00080000:2.0:1734019343.334475:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019343.334478:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019343.334490:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019343.334501:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247333139584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019343.334511:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019343.334515:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019343.334517:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019343.334520:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3605 00000100:00080000:2.0:1734019343.334523:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019343.334526:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019343.334531:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734019343.334541:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247333139712/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019343.334549:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019343.334552:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019343.334554:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019343.334557:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3605 00000100:00080000:2.0:1734019343.334560:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019343.334564:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019343.334568:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019343.334576:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333139840/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019343.334586:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019343.334589:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019343.334591:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019343.334596:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3605 00000100:00080000:2.0:1734019343.334599:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734019343.334600:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333139456:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734019343.334602:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019343.334606:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019343.334614:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333139968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019343.334622:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019343.334625:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019343.334627:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019343.334631:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3605 00000100:00080000:2.0:1734019343.334634:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019343.334637:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019343.334641:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019343.334649:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333140096/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019343.334657:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019343.334660:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019343.334661:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019343.334665:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3605 00000100:00080000:2.0:1734019343.334667:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019343.334670:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019343.334674:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019343.334682:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333140224/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734019343.334696:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019343.334723:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019343/real 1734019343] req@ffff8800a73ca680 x1818247333139456/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019343.334745:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ca680 x1818247333139456/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019343.334758:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019343.334762:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019343.334768:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333139456:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019343.334776:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333139584:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019343.334796:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019343.334802:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019343/real 1734019343] req@ffff8800a73c9f80 x1818247333139584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019343.334813:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9f80 x1818247333139584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019343.334821:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019343.334825:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019343.334829:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333139584:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019343.334836:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333139712:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019343.334852:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019343.334857:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019343/real 1734019343] req@ffff8800a73caa00 x1818247333139712/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019343.334867:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73caa00 x1818247333139712/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019343.334876:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019343.334878:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019343.334882:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333139712:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019343.334912:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333139840:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019343.334925:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019343.334931:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019343/real 1734019343] req@ffff8800a73cad80 x1818247333139840/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019343.334942:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333139840/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019343.334950:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019343.334953:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019343.334956:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333139840:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019343.334961:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333139968:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019343.334978:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019343.334984:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019343/real 1734019343] req@ffff8800a73cb100 x1818247333139968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019343.334995:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333139968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019343.335002:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019343.335005:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019343.335008:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333139968:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019343.335013:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333140096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019343.335026:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019343.335031:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019343/real 1734019343] req@ffff8800a73c9880 x1818247333140096/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019343.335042:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333140096/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019343.335049:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019343.335052:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019343.335055:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333140096:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019343.335061:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333140224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019343.335074:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019343.335079:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019343/real 1734019343] req@ffff8800a73c8e00 x1818247333140224/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019343.335096:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333140224/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019343.335104:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019343.335106:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019343.335109:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333140224:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019343.388483:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019344.390460:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019345.392494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019346.394501:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019347.396465:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019348.342330:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019348.342341:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019348.342345:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019348.342357:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3610 00000100:00080000:2.0:1734019348.342367:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019348.342375:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734019348.342411:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247333140352/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019348.342447:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019348.342451:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019348.342453:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019348.342458:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3610 00000100:00080000:2.0:1734019348.342461:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019348.342465:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019348.342475:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019348.342486:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247333140480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019348.342496:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019348.342499:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019348.342501:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019348.342504:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3610 00000100:00080000:2.0:1734019348.342507:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019348.342511:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019348.342516:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734019348.342524:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247333140608/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019348.342534:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019348.342537:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019348.342538:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019348.342542:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3610 00000100:00080000:2.0:1734019348.342544:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019348.342547:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019348.342552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019348.342562:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333140736/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019348.342571:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019348.342574:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019348.342576:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019348.342580:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3610 00000100:00080000:2.0:1734019348.342582:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734019348.342584:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333140352:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734019348.342586:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019348.342590:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019348.342598:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333140864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019348.342607:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019348.342610:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019348.342613:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019348.342617:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3610 00000100:00080000:2.0:1734019348.342620:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019348.342623:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019348.342628:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019348.342635:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333140992/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019348.342643:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019348.342646:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019348.342648:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019348.342651:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3610 00000100:00080000:2.0:1734019348.342654:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019348.342657:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019348.342661:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019348.342668:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333141120/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734019348.342677:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019348.342706:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019348/real 1734019348] req@ffff8800a73c8000 x1818247333140352/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019348.342728:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247333140352/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019348.342742:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019348.342746:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019348.342751:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333140352:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019348.342759:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333140480:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019348.342779:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019348.342786:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019348/real 1734019348] req@ffff8800a73c9500 x1818247333140480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019348.342798:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247333140480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019348.342806:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019348.342810:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019348.342813:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333140480:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019348.342820:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333140608:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019348.342836:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019348.342842:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019348/real 1734019348] req@ffff8800a73c9c00 x1818247333140608/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019348.342852:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247333140608/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019348.342860:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019348.342863:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019348.342867:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333140608:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019348.342897:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333140736:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019348.342910:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019348.342916:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019348/real 1734019348] req@ffff8800a73c8e00 x1818247333140736/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019348.342927:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333140736/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019348.342936:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019348.342938:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019348.342941:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333140736:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019348.342947:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333140864:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019348.342964:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019348.342970:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019348/real 1734019348] req@ffff8800a73c9880 x1818247333140864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019348.342981:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333140864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019348.342988:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019348.342990:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019348.342993:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333140864:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019348.342999:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333140992:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019348.343012:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019348.343018:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019348/real 1734019348] req@ffff8800a73cb100 x1818247333140992/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019348.343029:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333140992/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019348.343036:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019348.343039:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019348.343042:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333140992:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019348.343048:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333141120:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019348.343061:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019348.343118:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019348/real 1734019348] req@ffff8800a73cad80 x1818247333141120/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019348.343138:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333141120/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019348.343146:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019348.343148:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019348.343151:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333141120:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019348.398478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019349.400467:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019350.402449:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019351.404484:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019352.406407:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019353.350376:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019353.350387:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019353.350392:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019353.350403:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3615 00000100:00080000:2.0:1734019353.350413:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019353.350421:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734019353.350458:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73caa00 x1818247333141248/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019353.350499:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019353.350503:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019353.350505:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019353.350509:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3615 00000100:00080000:2.0:1734019353.350513:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019353.350517:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019353.350527:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019353.350538:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9f80 x1818247333141376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019353.350547:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019353.350550:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019353.350552:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019353.350555:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3615 00000100:00080000:2.0:1734019353.350558:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019353.350561:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019353.350566:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734019353.350575:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ca680 x1818247333141504/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019353.350584:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019353.350587:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019353.350588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019353.350592:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3615 00000100:00080000:2.0:1734019353.350594:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019353.350598:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019353.350603:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019353.350610:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333141248:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019353.350612:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333141632/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019353.350620:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019353.350624:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019353.350625:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019353.350628:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3615 00000100:00080000:2.0:1734019353.350631:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019353.350634:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019353.350639:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019353.350646:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333141760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019353.350654:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019353.350657:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019353.350658:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019353.350662:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3615 00000100:00080000:2.0:1734019353.350674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734019353.350678:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:2.0:1734019353.350678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019353.350683:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019353.350691:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333141888/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019353.350699:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019353/real 1734019353] req@ffff8800a73caa00 x1818247333141248/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019353.350699:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019353.350703:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019353.350705:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019353.350708:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3615 00000100:00080000:2.0:1734019353.350711:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019353.350715:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019353.350719:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73caa00 x1818247333141248/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:2.0:1734019353.350719:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019353.350726:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333142016/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019353.350729:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019353.350732:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019353.350737:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73caa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333141248:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019353.350745:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333141376:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019353.350763:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019353.350770:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019353/real 1734019353] req@ffff8800a73c9f80 x1818247333141376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019353.350781:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9f80 x1818247333141376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019353.350789:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019353.350792:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019353.350796:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333141376:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019353.350802:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333141504:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019353.350816:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019353.350821:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019353/real 1734019353] req@ffff8800a73ca680 x1818247333141504/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019353.350832:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ca680 x1818247333141504/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019353.350840:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019353.350842:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019353.350845:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ca680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333141504:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019353.350874:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333141632:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019353.350886:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019353.350892:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019353/real 1734019353] req@ffff8800a73cad80 x1818247333141632/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019353.350902:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333141632/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019353.350910:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019353.350912:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019353.350915:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333141632:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019353.350920:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333141760:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019353.350936:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019353.350941:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019353/real 1734019353] req@ffff8800a73cb100 x1818247333141760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019353.350952:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333141760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019353.350959:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019353.350961:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019353.350964:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333141760:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019353.350970:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333141888:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019353.351036:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019353.351044:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019353/real 1734019353] req@ffff8800a73c9880 x1818247333141888/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019353.351055:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333141888/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019353.351063:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019353.351065:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019353.351068:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333141888:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019353.351073:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333142016:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019353.351087:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019353.351094:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019353/real 1734019353] req@ffff8800a73c8e00 x1818247333142016/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019353.351110:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333142016/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019353.351118:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019353.351120:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019353.351123:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333142016:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019353.408504:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019354.410486:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019355.412499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019356.414449:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019357.416489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1734019358.358363:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019358.358375:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1734019358.358380:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019358.358391:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3620 00000100:00080000:2.0:1734019358.358403:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019358.358411:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:2.0:1734019358.358450:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9c00 x1818247333142144/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019358.358486:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019358.358490:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019358.358492:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019358.358497:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3620 00000100:00080000:2.0:1734019358.358500:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019358.358504:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019358.358515:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019358.358525:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9500 x1818247333142272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019358.358536:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019358.358540:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019358.358541:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019358.358545:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3620 00000100:00080000:2.0:1734019358.358547:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019358.358551:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019358.358556:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734019358.358564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8000 x1818247333142400/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019358.358573:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019358.358576:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019358.358578:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019358.358581:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3620 00000100:00080000:2.0:1734019358.358583:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019358.358587:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019358.358591:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019358.358600:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c8e00 x1818247333142528/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019358.358610:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019358.358614:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1734019358.358615:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019358.358619:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3620 00000100:00100000:0.0:1734019358.358621:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333142144:192.168.201.152@tcp:250:kworker.0 00000100:00080000:2.0:1734019358.358623:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019358.358626:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019358.358631:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1734019358.358639:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73c9880 x1818247333142656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019358.358647:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019358.358650:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1734019358.358652:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019358.358656:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3620 00000100:00080000:2.0:1734019358.358659:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019358.358662:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019358.358666:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019358.358674:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cb100 x1818247333142784/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019358.358682:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1734019358.358685:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1734019358.358687:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1734019358.358689:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3620 00000100:00080000:2.0:1734019358.358704:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019358.358708:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:2.0:1734019358.358712:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019358.358720:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cad80 x1818247333142912/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019358.358743:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333142272:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019358.358768:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333142400:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019358.358798:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333142528:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019358.358815:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333142656:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019358.358834:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333142784:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019358.358851:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333142912:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019358.418531:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019359.420519:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019360.422468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1734019361.364514:0:1957:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:1.0:1734019361.364563:0:1957:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1734019361.364582:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019361.364617:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019361.364626:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019361.364633:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019361.364640:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019361.364646:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019361.364653:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:2.0:1734019361.364730:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019358/real 1734019361] req@ffff8800a73c8e00 x1818247333142528/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019361.364767:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8e00 x1818247333142528/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019361.364785:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019361.364791:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019361.364799:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333142528:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019361.364811:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019358/real 1734019361] req@ffff8800a73c9880 x1818247333142656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019361.364831:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9880 x1818247333142656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019361.364896:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019361.364900:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019361.364906:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333142656:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019361.364918:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019358/real 1734019361] req@ffff8800a73cb100 x1818247333142784/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019361.364937:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cb100 x1818247333142784/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019361.364949:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019361.364953:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019361.364959:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333142784:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019361.364971:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019358/real 1734019361] req@ffff8800a73cad80 x1818247333142912/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019361.365002:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cad80 x1818247333142912/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019361.365015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019361.365019:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019361.365024:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333142912:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019361.365034:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019358/real 1734019361] req@ffff8800a73c9c00 x1818247333142144/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019361.365052:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9c00 x1818247333142144/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019361.365064:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019361.365068:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019361.365073:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333142144:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019361.365083:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019358/real 1734019361] req@ffff8800a73c9500 x1818247333142272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019361.365102:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c9500 x1818247333142272/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019361.365114:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019361.365118:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019361.365124:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333142272:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019361.365133:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019358/real 1734019361] req@ffff8800a73c8000 x1818247333142400/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019361.365151:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73c8000 x1818247333142400/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019361.365164:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019361.365168:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019361.365174:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333142400:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019361.424456:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019362.426494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019363.366449:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019363.366459:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019363.366464:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019363.366474:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3625 00000100:00080000:3.0:1734019363.366484:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019363.366491:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019363.366528:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333143040/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019363.366565:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019363.366570:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019363.366572:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019363.366576:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3625 00000100:00080000:3.0:1734019363.366580:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019363.366584:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019363.366593:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019363.366604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333143168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019363.366614:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019363.366617:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019363.366619:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019363.366624:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3625 00000100:00080000:3.0:1734019363.366627:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019363.366630:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019363.366635:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019363.366643:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333143296/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019363.366651:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019363.366654:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019363.366656:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019363.366659:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3625 00000100:00080000:3.0:1734019363.366662:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019363.366665:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:2.0:1734019363.366669:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333143040:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734019363.366670:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019363.366677:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333143424/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019363.366686:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019363.366689:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019363.366691:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019363.366694:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3625 00000100:00080000:3.0:1734019363.366697:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019363.366700:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019363.366704:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019363.366712:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333143552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019363.366721:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019363.366724:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019363.366726:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019363.366730:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3625 00000100:00080000:3.0:1734019363.366732:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734019363.366735:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019363.366736:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019363.366740:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019363.366747:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333143680/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019363.366755:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734019363.366757:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019363/real 1734019363] req@ffff8800a73cf480 x1818247333143040/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019363.366758:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019363.366759:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019363.366763:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3625 00000100:00080000:3.0:1734019363.366765:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019363.366769:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019363.366773:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019363.366777:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333143040/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019363.366782:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333143808/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019363.366788:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019363.366791:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019363.366796:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333143040:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019363.366803:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333143168:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019363.366855:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019363.366863:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019363/real 1734019363] req@ffff8800a73cd500 x1818247333143168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019363.366875:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333143168/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019363.366883:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019363.366887:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019363.366890:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333143168:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019363.366897:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333143296:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019363.366910:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019363.366916:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019363/real 1734019363] req@ffff8800a73cca80 x1818247333143296/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019363.366926:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333143296/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019363.366934:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019363.366936:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019363.366940:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333143296:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019363.366964:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333143424:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019363.366976:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019363.366982:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019363/real 1734019363] req@ffff8800a73cea00 x1818247333143424/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019363.366992:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333143424/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019363.367000:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019363.367002:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019363.367005:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333143424:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019363.367010:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333143552:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019363.367025:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019363.367031:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019363/real 1734019363] req@ffff8800a73cd880 x1818247333143552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019363.367041:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333143552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019363.367048:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019363.367050:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019363.367054:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333143552:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019363.367058:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333143680:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019363.367069:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019363.367075:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019363/real 1734019363] req@ffff8800a73ced80 x1818247333143680/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019363.367085:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333143680/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019363.367092:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019363.367094:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019363.367097:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333143680:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019363.367101:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333143808:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019363.367113:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019363.367119:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019363/real 1734019363] req@ffff8800a73cce00 x1818247333143808/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019363.367135:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333143808/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019363.367143:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019363.367145:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019363.367148:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333143808:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019363.428413:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019364.430483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019365.432483:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019366.434495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019367.436467:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019368.374336:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019368.374348:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019368.374353:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019368.374363:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3630 00000100:00080000:3.0:1734019368.374372:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019368.374380:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019368.374417:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333143936/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019368.374452:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019368.374456:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019368.374458:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019368.374463:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3630 00000100:00080000:3.0:1734019368.374466:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019368.374470:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019368.374480:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019368.374492:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333144064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019368.374502:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019368.374505:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019368.374507:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019368.374511:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3630 00000100:00080000:3.0:1734019368.374514:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019368.374517:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019368.374522:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019368.374530:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333144192/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019368.374540:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019368.374543:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019368.374545:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019368.374548:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3630 00000100:00080000:3.0:1734019368.374551:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019368.374554:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019368.374558:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019368.374565:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333143936:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019368.374566:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333144320/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019368.374575:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019368.374578:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019368.374580:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019368.374583:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3630 00000100:00080000:3.0:1734019368.374586:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019368.374589:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019368.374593:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019368.374601:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333144448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019368.374609:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019368.374612:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019368.374613:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019368.374618:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3630 00000100:00080000:3.0:1734019368.374621:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019368.374625:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019368.374629:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019368.374631:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019368.374637:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333144576/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019368.374645:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019368.374648:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019368.374649:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019368.374652:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019368/real 1734019368] req@ffff8800a73ce680 x1818247333143936/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019368.374653:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3630 00000100:00080000:3.0:1734019368.374656:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019368.374659:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019368.374663:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019368.374671:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333143936/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019368.374671:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333144704/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019368.374682:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019368.374685:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019368.374690:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333143936:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019368.374697:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333144064:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019368.374714:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019368.374720:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019368/real 1734019368] req@ffff8800a73cce00 x1818247333144064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019368.374778:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333144064/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019368.374786:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019368.374790:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019368.374794:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333144064:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019368.374801:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333144192:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019368.374819:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019368.374825:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019368/real 1734019368] req@ffff8800a73ced80 x1818247333144192/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019368.374836:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333144192/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019368.374844:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019368.374846:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019368.374850:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333144192:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019368.374873:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333144320:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019368.374886:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019368.374891:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019368/real 1734019368] req@ffff8800a73cd880 x1818247333144320/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019368.374902:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333144320/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019368.374910:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019368.374912:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019368.374915:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333144320:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019368.374920:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333144448:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019368.374935:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019368.374941:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019368/real 1734019368] req@ffff8800a73cea00 x1818247333144448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019368.374951:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333144448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019368.374959:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019368.374961:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019368.374964:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333144448:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019368.374968:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333144576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019368.374980:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019368.374986:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019368/real 1734019368] req@ffff8800a73cca80 x1818247333144576/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019368.374996:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333144576/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019368.375004:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019368.375006:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019368.375009:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333144576:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019368.375014:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333144704:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019368.375026:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019368.375032:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019368/real 1734019368] req@ffff8800a73cd500 x1818247333144704/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019368.375049:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333144704/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019368.375056:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019368.375059:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019368.375061:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333144704:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019368.438493:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019369.440511:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019370.442496:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019371.444483:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019372.446433:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019373.382308:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019373.382315:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019373.382318:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019373.382325:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3635 00000100:00080000:3.0:1734019373.382332:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019373.382336:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019373.382359:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333144832/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019373.382379:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019373.382381:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019373.382382:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019373.382399:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3635 00000100:00080000:3.0:1734019373.382401:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019373.382403:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019373.382409:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019373.382414:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333144960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019373.382419:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019373.382421:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019373.382422:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019373.382424:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3635 00000100:00080000:3.0:1734019373.382425:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019373.382426:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019373.382428:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019373.382432:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333145088/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019373.382436:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019373.382437:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019373.382438:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019373.382440:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3635 00000100:00080000:3.0:1734019373.382441:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019373.382442:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019373.382444:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019373.382447:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333145216/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019373.382451:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019373.382452:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019373.382453:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019373.382454:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3635 00000100:00080000:3.0:1734019373.382455:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019373.382456:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019373.382458:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019373.382462:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333145344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019373.382463:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333144832:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019373.382465:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019373.382467:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019373.382467:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019373.382469:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3635 00000100:00080000:3.0:1734019373.382470:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019373.382472:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019373.382473:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019373.382476:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333145472/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019373.382480:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019373.382482:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019373.382483:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019373.382484:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3635 00000100:00080000:3.0:1734019373.382485:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019373.382487:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019373.382489:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019373.382493:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333145600/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:2.0:1734019373.382503:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019373.382516:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019373/real 1734019373] req@ffff8800a73cf480 x1818247333144832/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019373.382525:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333144832/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019373.382530:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019373.382531:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019373.382533:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333144832:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019373.382536:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333144960:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019373.382544:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019373.382547:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019373/real 1734019373] req@ffff8800a73cc380 x1818247333144960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019373.382551:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333144960/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019373.382555:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019373.382557:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019373.382558:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333144960:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019373.382561:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333145088:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019373.382567:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019373.382569:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019373/real 1734019373] req@ffff8800a73cf100 x1818247333145088/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019373.382574:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333145088/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019373.382577:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019373.382578:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019373.382580:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333145088:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019373.382582:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333145216:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019373.382587:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019373.382589:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019373/real 1734019373] req@ffff8800a73cc700 x1818247333145216/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019373.382593:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333145216/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019373.382597:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019373.382598:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019373.382599:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333145216:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019373.382614:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333145344:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019373.382621:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019373.382623:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019373/real 1734019373] req@ffff8800a73cdf80 x1818247333145344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019373.382628:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333145344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019373.382632:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019373.382633:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019373.382634:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333145344:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019373.382636:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333145472:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019373.382686:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019373.382690:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019373/real 1734019373] req@ffff8800a73cf800 x1818247333145472/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019373.382694:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333145472/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019373.382697:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019373.382698:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019373.382700:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333145472:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019373.382702:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333145600:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019373.382707:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019373.382710:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019373/real 1734019373] req@ffff8800a73cfb80 x1818247333145600/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019373.382717:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333145600/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019373.382721:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019373.382722:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019373.382723:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333145600:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019373.448415:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019374.450421:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019375.452382:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019376.454489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019377.456501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019378.390348:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019378.390358:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019378.390363:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019378.390374:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3640 00000100:00080000:3.0:1734019378.390384:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019378.390392:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019378.390429:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333145728/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019378.390464:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019378.390469:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019378.390471:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019378.390476:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3640 00000100:00080000:3.0:1734019378.390479:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019378.390483:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019378.390493:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019378.390506:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333145856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019378.390516:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019378.390520:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019378.390522:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019378.390525:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3640 00000100:00080000:3.0:1734019378.390528:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019378.390532:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019378.390536:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019378.390545:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333145984/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019378.390586:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019378.390589:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019378.390591:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019378.390595:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3640 00000100:00080000:3.0:1734019378.390598:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019378.390601:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019378.390606:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019378.390615:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333146112/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019378.390622:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333145728:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019378.390625:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019378.390629:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019378.390630:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019378.390634:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3640 00000100:00080000:3.0:1734019378.390636:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019378.390640:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019378.390644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019378.390652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333146240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019378.390660:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019378.390663:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019378.390665:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019378.390669:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3640 00000100:00080000:3.0:1734019378.390680:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019378.390684:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:2.0:1734019378.390689:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734019378.390689:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019378.390697:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333146368/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019378.390705:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019378.390708:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019378.390710:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019378.390711:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019378/real 1734019378] req@ffff8800a73cdc00 x1818247333145728/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019378.390713:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3640 00000100:00080000:3.0:1734019378.390717:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019378.390720:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019378.390724:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019378.390730:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333145728/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019378.390733:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333146496/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019378.390741:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019378.390744:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019378.390749:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333145728:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019378.390756:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333145856:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019378.390773:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019378.390780:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019378/real 1734019378] req@ffff8800a73cfb80 x1818247333145856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019378.390791:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333145856/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019378.390798:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019378.390802:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019378.390805:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333145856:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019378.390812:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333145984:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019378.390825:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019378.390831:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019378/real 1734019378] req@ffff8800a73cf800 x1818247333145984/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019378.390841:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333145984/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019378.390849:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019378.390852:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019378.390855:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333145984:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019378.390880:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333146112:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019378.390892:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019378.390898:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019378/real 1734019378] req@ffff8800a73cdf80 x1818247333146112/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019378.390908:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333146112/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019378.390916:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019378.390918:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019378.390921:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333146112:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019378.390926:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333146240:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019378.390941:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019378.390947:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019378/real 1734019378] req@ffff8800a73cc700 x1818247333146240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019378.390957:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333146240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019378.390964:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019378.390967:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019378.390970:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333146240:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019378.390974:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333146368:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019378.390986:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019378.390992:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019378/real 1734019378] req@ffff8800a73cf100 x1818247333146368/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019378.391002:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333146368/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019378.391009:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019378.391011:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019378.391014:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333146368:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019378.391019:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333146496:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019378.391031:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019378.391036:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019378/real 1734019378] req@ffff8800a73cc380 x1818247333146496/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019378.391053:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333146496/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019378.391061:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019378.391063:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019378.391066:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333146496:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019378.458475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019379.460507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019380.462478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019381.464526:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019382.466451:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019383.398368:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019383.398379:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019383.398383:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019383.398394:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3645 00000100:00080000:3.0:1734019383.398404:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019383.398412:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019383.398449:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333146624/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019383.398527:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019383.398532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019383.398534:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019383.398539:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3645 00000100:00080000:3.0:1734019383.398542:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019383.398547:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019383.398557:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019383.398570:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333146752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019383.398580:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019383.398584:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019383.398585:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019383.398589:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3645 00000100:00080000:3.0:1734019383.398592:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019383.398595:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019383.398600:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019383.398608:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333146880/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019383.398616:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019383.398619:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019383.398621:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019383.398624:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3645 00000100:00080000:3.0:1734019383.398627:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019383.398630:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019383.398634:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019383.398642:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333147008/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019383.398646:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333146624:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019383.398651:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019383.398654:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019383.398656:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019383.398660:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3645 00000100:00080000:3.0:1734019383.398663:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019383.398666:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019383.398670:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019383.398677:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333147136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019383.398686:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019383.398689:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019383.398690:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019383.398694:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3645 00000100:00080000:3.0:1734019383.398706:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019383.398709:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:2.0:1734019383.398713:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734019383.398714:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019383.398721:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333147264/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019383.398729:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019383.398732:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019383.398734:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019383.398735:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019383/real 1734019383] req@ffff8800a73cf480 x1818247333146624/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019383.398737:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3645 00000100:00080000:3.0:1734019383.398740:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019383.398743:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019383.398748:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019383.398755:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333146624/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019383.398757:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333147392/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019383.398766:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019383.398769:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019383.398774:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333146624:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019383.398781:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333146752:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019383.398798:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019383.398804:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019383/real 1734019383] req@ffff8800a73cd500 x1818247333146752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019383.398815:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333146752/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019383.398823:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019383.398827:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019383.398831:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333146752:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019383.398837:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333146880:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019383.398850:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019383.398856:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019383/real 1734019383] req@ffff8800a73cca80 x1818247333146880/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019383.398866:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333146880/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019383.398874:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019383.398876:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019383.398880:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333146880:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019383.398904:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333147008:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019383.398917:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019383.398923:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019383/real 1734019383] req@ffff8800a73cea00 x1818247333147008/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019383.398933:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333147008/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019383.398941:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019383.398943:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019383.398946:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333147008:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019383.398950:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333147136:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019383.398965:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019383.398971:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019383/real 1734019383] req@ffff8800a73cd880 x1818247333147136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019383.398981:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333147136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019383.398989:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019383.398991:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019383.398993:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333147136:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019383.398998:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333147264:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019383.399010:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019383.399015:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019383/real 1734019383] req@ffff8800a73ced80 x1818247333147264/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019383.399025:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333147264/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019383.399032:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019383.399034:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019383.399037:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333147264:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019383.399042:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333147392:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019383.399054:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019383.399059:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019383/real 1734019383] req@ffff8800a73cce00 x1818247333147392/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019383.399086:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333147392/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019383.399094:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019383.399096:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019383.399099:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333147392:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019383.468531:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019384.470531:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019385.472494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019386.474524:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019387.476485:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019388.406368:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019388.406379:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019388.406417:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019388.406429:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3650 00000100:00080000:3.0:1734019388.406438:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019388.406446:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019388.406485:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333147520/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019388.406519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019388.406524:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019388.406526:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019388.406530:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3650 00000100:00080000:3.0:1734019388.406533:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019388.406537:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019388.406547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019388.406560:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333147648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019388.406570:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019388.406574:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019388.406576:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019388.406580:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3650 00000100:00080000:3.0:1734019388.406583:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019388.406586:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019388.406591:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019388.406599:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333147776/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019388.406608:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019388.406611:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019388.406612:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019388.406615:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3650 00000100:00080000:3.0:1734019388.406618:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019388.406621:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019388.406626:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019388.406633:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333147904/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019388.406641:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019388.406645:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019388.406646:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019388.406650:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3650 00000100:00080000:3.0:1734019388.406652:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734019388.406653:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333147520:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019388.406655:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019388.406659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019388.406666:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333148032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019388.406675:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019388.406679:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019388.406680:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019388.406684:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3650 00000100:00080000:3.0:1734019388.406687:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019388.406690:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019388.406694:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019388.406701:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333148160/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019388.406709:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019388.406712:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019388.406714:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019388.406717:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3650 00000100:00080000:3.0:1734019388.406720:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734019388.406722:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019388.406723:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019388.406727:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019388.406735:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333148288/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019388.406745:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019388/real 1734019388] req@ffff8800a73ce680 x1818247333147520/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019388.406765:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333147520/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019388.406777:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019388.406780:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019388.406785:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333147520:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019388.406792:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333147648:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019388.406809:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019388.406815:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019388/real 1734019388] req@ffff8800a73cce00 x1818247333147648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019388.406826:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333147648/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019388.406834:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019388.406838:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019388.406841:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333147648:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019388.406848:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333147776:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019388.406862:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019388.406868:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019388/real 1734019388] req@ffff8800a73ced80 x1818247333147776/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019388.406878:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333147776/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019388.406886:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019388.406889:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019388.406892:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333147776:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019388.406916:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333147904:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019388.406928:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019388.406934:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019388/real 1734019388] req@ffff8800a73cd880 x1818247333147904/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019388.406945:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333147904/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019388.406953:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019388.406955:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019388.406958:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333147904:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019388.406963:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333148032:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019388.406978:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019388.406984:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019388/real 1734019388] req@ffff8800a73cea00 x1818247333148032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019388.406994:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333148032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019388.407001:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019388.407003:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019388.407006:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333148032:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019388.407011:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333148160:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019388.407022:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019388.407027:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019388/real 1734019388] req@ffff8800a73cca80 x1818247333148160/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019388.407037:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333148160/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019388.407044:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019388.407046:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019388.407049:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333148160:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019388.407054:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333148288:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019388.407066:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019388.407071:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019388/real 1734019388] req@ffff8800a73cd500 x1818247333148288/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019388.407087:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333148288/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019388.407095:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019388.407097:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019388.407100:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333148288:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019388.478466:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019389.480472:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019390.482514:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019391.484525:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019392.486532:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019393.414384:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019393.414394:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019393.414399:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019393.414410:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3655 00000100:00080000:3.0:1734019393.414419:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019393.414427:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019393.414463:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333148416/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019393.414496:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019393.414500:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019393.414502:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019393.414507:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3655 00000100:00080000:3.0:1734019393.414510:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019393.414514:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019393.414524:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019393.414534:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333148544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019393.414544:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019393.414547:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019393.414549:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019393.414553:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3655 00000100:00080000:3.0:1734019393.414555:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019393.414559:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019393.414563:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019393.414571:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333148672/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019393.414581:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019393.414584:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019393.414586:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019393.414589:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3655 00000100:00080000:3.0:1734019393.414591:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019393.414594:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019393.414599:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019393.414606:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333148800/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019393.414614:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734019393.414616:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333148416:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019393.414618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019393.414619:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019393.414623:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3655 00000100:00080000:3.0:1734019393.414625:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019393.414628:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019393.414633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019393.414640:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333148928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019393.414648:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019393.414651:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019393.414653:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019393.414656:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3655 00000100:00080000:3.0:1734019393.414675:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019393.414679:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019393.414683:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019393.414684:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019393.414691:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333149056/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019393.414699:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019393.414702:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019393.414704:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019393.414706:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019393/real 1734019393] req@ffff8800a73cf480 x1818247333148416/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019393.414707:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3655 00000100:00080000:3.0:1734019393.414710:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019393.414713:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019393.414717:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019393.414725:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333148416/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019393.414727:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333149184/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019393.414736:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019393.414740:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019393.414745:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333148416:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019393.414752:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333148544:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019393.414769:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019393.414776:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019393/real 1734019393] req@ffff8800a73cc380 x1818247333148544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019393.414787:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333148544/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019393.414795:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019393.414798:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019393.414802:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333148544:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019393.414808:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333148672:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019393.414821:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019393.414827:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019393/real 1734019393] req@ffff8800a73cf100 x1818247333148672/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019393.414837:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333148672/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019393.414845:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019393.414848:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019393.414851:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333148672:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019393.414875:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333148800:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019393.414887:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019393.414893:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019393/real 1734019393] req@ffff8800a73cc700 x1818247333148800/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019393.414903:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333148800/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019393.414911:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019393.414913:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019393.414916:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333148800:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019393.414921:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333148928:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019393.414936:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019393.414942:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019393/real 1734019393] req@ffff8800a73cdf80 x1818247333148928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019393.414952:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333148928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019393.414959:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019393.414962:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019393.414965:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333148928:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019393.414969:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333149056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019393.414981:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019393.414986:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019393/real 1734019393] req@ffff8800a73cf800 x1818247333149056/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019393.414996:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333149056/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019393.415003:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019393.415005:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019393.415008:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333149056:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019393.415013:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333149184:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019393.415024:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019393.415030:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019393/real 1734019393] req@ffff8800a73cfb80 x1818247333149184/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019393.415046:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333149184/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019393.415053:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019393.415055:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019393.415058:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333149184:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019393.488491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019394.490522:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019395.492501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019396.494507:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019397.496482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019398.422328:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019398.422338:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019398.422343:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019398.422354:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3660 00000100:00080000:3.0:1734019398.422363:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019398.422371:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019398.422409:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333149312/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019398.422445:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019398.422449:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019398.422451:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019398.422455:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3660 00000100:00080000:3.0:1734019398.422459:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019398.422463:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019398.422472:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019398.422485:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333149440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019398.422503:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019398.422506:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019398.422508:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019398.422512:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3660 00000100:00080000:3.0:1734019398.422515:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019398.422518:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019398.422524:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019398.422532:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333149568/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019398.422540:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019398.422543:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019398.422545:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019398.422549:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3660 00000100:00080000:3.0:1734019398.422552:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019398.422555:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019398.422559:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019398.422567:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333149696/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019398.422571:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333149312:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019398.422576:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019398.422579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019398.422581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019398.422584:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3660 00000100:00080000:3.0:1734019398.422587:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019398.422590:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019398.422594:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019398.422602:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333149824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019398.422610:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019398.422613:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019398.422614:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019398.422618:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3660 00000100:00080000:3.0:1734019398.422630:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019398.422634:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019398.422638:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019398.422639:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019398.422646:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333149952/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019398.422655:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019398.422658:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019398.422660:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019398.422661:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019398/real 1734019398] req@ffff8800a73cdc00 x1818247333149312/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019398.422663:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3660 00000100:00080000:3.0:1734019398.422666:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019398.422669:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019398.422673:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019398.422680:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333149312/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019398.422682:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333150080/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019398.422692:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019398.422695:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019398.422700:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333149312:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019398.422707:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333149440:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019398.422724:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019398.422730:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019398/real 1734019398] req@ffff8800a73cfb80 x1818247333149440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019398.422741:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333149440/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019398.422749:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019398.422753:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019398.422756:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333149440:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019398.422763:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333149568:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019398.422776:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019398.422782:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019398/real 1734019398] req@ffff8800a73cf800 x1818247333149568/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019398.422792:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333149568/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019398.422801:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019398.422803:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019398.422807:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333149568:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019398.422832:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333149696:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019398.422844:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019398.422849:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019398/real 1734019398] req@ffff8800a73cdf80 x1818247333149696/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019398.422860:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333149696/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019398.422869:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019398.422871:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019398.422874:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333149696:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019398.422879:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333149824:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019398.422895:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019398.422901:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019398/real 1734019398] req@ffff8800a73cc700 x1818247333149824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019398.422911:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333149824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019398.422919:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019398.422921:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019398.422924:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333149824:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019398.422930:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333149952:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019398.422941:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019398.422946:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019398/real 1734019398] req@ffff8800a73cf100 x1818247333149952/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019398.422956:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333149952/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019398.422963:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019398.422965:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019398.422968:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333149952:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019398.422974:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333150080:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019398.422986:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019398.422992:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019398/real 1734019398] req@ffff8800a73cc380 x1818247333150080/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019398.423008:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333150080/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019398.423015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019398.423018:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019398.423020:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333150080:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019398.498484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019399.500388:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019400.502462:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019401.504469:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019402.506490:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019403.430325:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019403.430335:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019403.430340:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019403.430350:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3665 00000100:00080000:3.0:1734019403.430359:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019403.430368:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019403.430403:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333150208/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019403.430437:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019403.430441:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019403.430443:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019403.430448:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3665 00000100:00080000:3.0:1734019403.430451:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019403.430455:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019403.430465:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019403.430476:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333150336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019403.430485:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019403.430489:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019403.430491:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019403.430494:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3665 00000100:00080000:3.0:1734019403.430497:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019403.430501:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019403.430506:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019403.430513:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333150464/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019403.430522:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019403.430525:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019403.430526:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019403.430530:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3665 00000100:00080000:3.0:1734019403.430532:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019403.430536:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019403.430540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019403.430548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333150592/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019403.430556:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333150208:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019403.430557:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019403.430561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019403.430563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019403.430567:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3665 00000100:00080000:3.0:1734019403.430569:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019403.430573:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019403.430577:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019403.430584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333150720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019403.430592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019403.430595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019403.430597:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019403.430601:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3665 00000100:00080000:3.0:1734019403.430613:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019403.430617:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019403.430622:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019403.430625:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019403.430629:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333150848/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019403.430637:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019403.430640:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019403.430642:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019403.430645:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3665 00000100:00100000:2.0:1734019403.430648:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019403/real 1734019403] req@ffff8800a73cf480 x1818247333150208/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019403.430649:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019403.430652:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019403.430657:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019403.430665:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333150976/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019403.430668:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333150208/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019403.430678:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019403.430681:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019403.430686:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333150208:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019403.430693:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333150336:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019403.430710:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019403.430716:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019403/real 1734019403] req@ffff8800a73cd500 x1818247333150336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019403.430727:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333150336/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019403.430735:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019403.430738:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019403.430742:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333150336:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019403.430748:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333150464:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019403.430761:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019403.430767:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019403/real 1734019403] req@ffff8800a73cca80 x1818247333150464/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019403.430777:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333150464/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019403.430784:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019403.430787:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019403.430790:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333150464:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019403.430815:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333150592:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019403.430827:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019403.430833:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019403/real 1734019403] req@ffff8800a73cea00 x1818247333150592/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019403.430844:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333150592/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019403.430851:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019403.430853:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019403.430856:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333150592:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019403.430861:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333150720:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019403.430876:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019403.430882:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019403/real 1734019403] req@ffff8800a73cd880 x1818247333150720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019403.430892:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333150720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019403.430899:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019403.430901:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019403.430904:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333150720:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019403.430909:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333150848:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019403.430920:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019403.430926:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019403/real 1734019403] req@ffff8800a73ced80 x1818247333150848/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019403.430935:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333150848/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019403.430943:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019403.430945:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019403.430948:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333150848:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019403.430953:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333150976:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019403.430965:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019403.430970:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019403/real 1734019403] req@ffff8800a73cce00 x1818247333150976/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019403.430986:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333150976/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019403.430994:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019403.430996:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019403.430999:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333150976:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019403.508466:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019404.510510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019405.512510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019406.514487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019407.516496:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019408.438377:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019408.438389:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019408.438393:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019408.438405:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3670 00000100:00080000:3.0:1734019408.438415:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019408.438423:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019408.438460:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333151104/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019408.438497:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019408.438501:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019408.438503:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019408.438508:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3670 00000100:00080000:3.0:1734019408.438511:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019408.438515:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019408.438525:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019408.438538:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333151232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019408.438548:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019408.438552:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019408.438553:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019408.438557:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3670 00000100:00080000:3.0:1734019408.438559:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019408.438563:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019408.438568:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019408.438576:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333151360/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019408.438584:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019408.438587:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019408.438588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019408.438592:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3670 00000100:00080000:3.0:1734019408.438594:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019408.438598:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019408.438602:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019408.438609:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333151488/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019408.438618:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333151104:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019408.438618:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019408.438622:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019408.438623:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019408.438627:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3670 00000100:00080000:3.0:1734019408.438630:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019408.438633:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019408.438637:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019408.438645:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333151616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019408.438653:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019408.438656:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019408.438657:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019408.438662:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3670 00000100:00080000:3.0:1734019408.438676:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019408.438680:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019408.438684:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019408.438686:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019408.438693:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333151744/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019408.438701:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019408.438704:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019408.438705:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019408.438708:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019408/real 1734019408] req@ffff8800a73ce680 x1818247333151104/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019408.438709:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3670 00000100:00080000:3.0:1734019408.438712:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019408.438715:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019408.438719:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019408.438727:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333151872/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019408.438728:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333151104/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019408.438739:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019408.438742:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019408.438747:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333151104:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019408.438754:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333151232:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019408.438771:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019408.438778:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019408/real 1734019408] req@ffff8800a73cce00 x1818247333151232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019408.438789:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333151232/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019408.438796:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019408.438800:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019408.438804:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333151232:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019408.438810:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333151360:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019408.438824:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019408.438829:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019408/real 1734019408] req@ffff8800a73ced80 x1818247333151360/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019408.438839:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333151360/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019408.438847:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019408.438850:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019408.438853:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333151360:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019408.438878:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333151488:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019408.438891:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019408.438897:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019408/real 1734019408] req@ffff8800a73cd880 x1818247333151488/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019408.438907:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333151488/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019408.438915:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019408.438917:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019408.438920:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333151488:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019408.438925:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333151616:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019408.438940:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019408.438946:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019408/real 1734019408] req@ffff8800a73cea00 x1818247333151616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019408.438956:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333151616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019408.438963:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019408.438965:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019408.438969:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333151616:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019408.438973:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333151744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019408.438984:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019408.438990:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019408/real 1734019408] req@ffff8800a73cca80 x1818247333151744/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019408.438999:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333151744/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019408.439006:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019408.439009:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019408.439012:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333151744:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019408.439016:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333151872:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019408.439028:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019408.439034:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019408/real 1734019408] req@ffff8800a73cd500 x1818247333151872/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019408.439106:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333151872/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019408.439114:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019408.439117:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019408.439120:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333151872:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019408.518480:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019409.520480:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019410.522476:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019411.524492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019412.526476:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019413.446365:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019413.446375:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019413.446379:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019413.446390:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3675 00000100:00080000:3.0:1734019413.446400:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019413.446408:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019413.446445:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333152000/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019413.446480:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019413.446484:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019413.446486:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019413.446490:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3675 00000100:00080000:3.0:1734019413.446494:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019413.446498:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019413.446507:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019413.446518:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333152128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019413.446528:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019413.446531:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019413.446534:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019413.446537:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3675 00000100:00080000:3.0:1734019413.446540:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019413.446544:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019413.446548:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019413.446557:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333152256/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019413.446565:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019413.446568:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019413.446569:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019413.446572:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3675 00000100:00080000:3.0:1734019413.446575:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019413.446578:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019413.446582:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019413.446589:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333152384/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019413.446598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734019413.446601:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333152000:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019413.446601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019413.446603:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019413.446606:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3675 00000100:00080000:3.0:1734019413.446609:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019413.446612:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019413.446616:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019413.446624:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333152512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019413.446634:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019413.446637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019413.446638:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019413.446642:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3675 00000100:00080000:3.0:1734019413.446657:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019413.446660:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019413.446665:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019413.446666:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019413.446673:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333152640/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019413.446681:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019413.446684:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019413.446686:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019413.446687:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019413/real 1734019413] req@ffff8800a73cf480 x1818247333152000/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019413.446689:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3675 00000100:00080000:3.0:1734019413.446692:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019413.446695:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019413.446699:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019413.446707:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333152000/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019413.446708:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333152768/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019413.446718:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019413.446721:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019413.446726:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333152000:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019413.446734:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333152128:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019413.446751:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019413.446757:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019413/real 1734019413] req@ffff8800a73cc380 x1818247333152128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019413.446768:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333152128/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019413.446776:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019413.446780:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019413.446783:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333152128:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019413.446789:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333152256:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019413.446803:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019413.446808:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019413/real 1734019413] req@ffff8800a73cf100 x1818247333152256/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019413.446818:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333152256/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019413.446826:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019413.446829:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019413.446832:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333152256:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019413.446855:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333152384:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019413.446867:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019413.446873:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019413/real 1734019413] req@ffff8800a73cc700 x1818247333152384/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019413.446884:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333152384/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019413.446892:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019413.446894:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019413.446897:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333152384:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019413.446902:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333152512:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019413.446917:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019413.446923:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019413/real 1734019413] req@ffff8800a73cdf80 x1818247333152512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019413.446932:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333152512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019413.446940:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019413.446942:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019413.446945:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333152512:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019413.446950:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333152640:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019413.447020:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019413.447029:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019413/real 1734019413] req@ffff8800a73cf800 x1818247333152640/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019413.447039:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333152640/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019413.447047:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019413.447049:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019413.447052:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333152640:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019413.447058:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333152768:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019413.447073:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019413.447078:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019413/real 1734019413] req@ffff8800a73cfb80 x1818247333152768/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019413.447095:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333152768/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019413.447103:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019413.447105:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019413.447108:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333152768:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019413.528488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019414.530447:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019415.532486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019416.534497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019417.536496:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019418.454381:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019418.454393:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019418.454397:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019418.454409:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3680 00000100:00080000:3.0:1734019418.454419:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019418.454427:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019418.454465:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333152896/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019418.454500:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019418.454504:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019418.454506:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019418.454511:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3680 00000100:00080000:3.0:1734019418.454514:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019418.454518:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019418.454528:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019418.454540:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333153024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019418.454550:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019418.454554:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019418.454556:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019418.454559:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3680 00000100:00080000:3.0:1734019418.454562:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019418.454565:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019418.454570:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019418.454578:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333153152/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019418.454587:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019418.454590:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019418.454592:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019418.454595:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3680 00000100:00080000:3.0:1734019418.454597:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019418.454601:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019418.454605:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019418.454612:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333153280/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019418.454621:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019418.454624:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019418.454625:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019418.454626:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333152896:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019418.454629:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3680 00000100:00080000:3.0:1734019418.454631:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019418.454635:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019418.454639:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019418.454647:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333153408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019418.454655:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019418.454657:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019418.454659:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019418.454663:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3680 00000100:00080000:3.0:1734019418.454683:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019418.454686:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019418.454691:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019418.454694:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019418.454699:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333153536/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019418.454707:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019418.454710:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019418.454712:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019418.454715:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019418/real 1734019418] req@ffff8800a73cdc00 x1818247333152896/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019418.454715:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3680 00000100:00080000:3.0:1734019418.454718:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019418.454721:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019418.454725:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019418.454733:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333153664/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019418.454734:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333152896/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019418.454745:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019418.454749:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019418.454753:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333152896:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019418.454760:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333153024:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019418.454777:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019418.454783:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019418/real 1734019418] req@ffff8800a73cfb80 x1818247333153024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019418.454794:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333153024/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019418.454802:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019418.454806:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019418.454809:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333153024:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019418.454815:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333153152:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019418.454829:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019418.454835:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019418/real 1734019418] req@ffff8800a73cf800 x1818247333153152/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019418.454845:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333153152/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019418.454852:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019418.454855:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019418.454858:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333153152:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019418.454930:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333153280:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019418.454947:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019418.454955:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019418/real 1734019418] req@ffff8800a73cdf80 x1818247333153280/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019418.454966:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333153280/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019418.454974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019418.454976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019418.454979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333153280:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019418.454984:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333153408:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019418.454999:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019418.455005:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019418/real 1734019418] req@ffff8800a73cc700 x1818247333153408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019418.455016:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333153408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019418.455023:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019418.455025:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019418.455028:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333153408:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019418.455033:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333153536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019418.455044:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019418.455050:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019418/real 1734019418] req@ffff8800a73cf100 x1818247333153536/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019418.455060:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333153536/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019418.455066:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019418.455068:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019418.455071:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333153536:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019418.455076:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333153664:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019418.455088:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019418.455094:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019418/real 1734019418] req@ffff8800a73cc380 x1818247333153664/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019418.455110:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333153664/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019418.455118:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019418.455121:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019418.455123:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333153664:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019418.538478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019419.540498:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019420.542503:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019421.544464:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019422.546453:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019423.462334:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019423.462347:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019423.462353:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019423.462368:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3686 00000100:00080000:3.0:1734019423.462378:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019423.462389:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019423.462440:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333153792/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019423.462478:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019423.462482:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019423.462484:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019423.462490:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3686 00000100:00080000:3.0:1734019423.462493:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019423.462498:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019423.462509:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019423.462520:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333153920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019423.462531:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019423.462534:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019423.462536:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019423.462540:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3686 00000100:00080000:3.0:1734019423.462543:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019423.462547:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019423.462552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019423.462561:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333154048/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019423.462569:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019423.462573:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019423.462574:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019423.462578:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3686 00000100:00080000:3.0:1734019423.462581:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019423.462585:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019423.462589:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019423.462598:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333154176/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019423.462607:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019423.462610:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019423.462612:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019423.462616:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3686 00000100:00080000:3.0:1734019423.462618:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734019423.462619:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333153792:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019423.462622:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019423.462626:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019423.462634:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333154304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019423.462642:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019423.462645:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019423.462647:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019423.462651:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3686 00000100:00080000:3.0:1734019423.462655:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019423.462658:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019423.462663:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019423.462671:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333154432/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019423.462680:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019423.462684:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019423.462685:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019423.462689:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3686 00000100:00080000:3.0:1734019423.462692:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019423.462695:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019423.462700:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019423.462710:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333154560/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019423.462729:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333153920:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019423.462752:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333154048:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019423.462777:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333154176:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019423.462826:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333154304:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019423.462845:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333154432:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019423.462861:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333154560:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019423.548496:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019424.550514:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019425.552474:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734019426.468475:0:1958:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734019426.468525:0:1958:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734019426.468543:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019426.468578:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019426.468586:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019426.468593:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019426.468599:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019426.468607:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019426.468614:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:3.0:1734019426.468681:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019423/real 1734019426] req@ffff8800a73cea00 x1818247333154176/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019426.468710:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333154176/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019426.468723:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734019426.468727:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734019426.468732:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333154176:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734019426.468740:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019423/real 1734019426] req@ffff8800a73cd880 x1818247333154304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019426.468804:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333154304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019426.468814:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734019426.468817:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734019426.468821:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333154304:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734019426.468827:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019423/real 1734019426] req@ffff8800a73ced80 x1818247333154432/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019426.468837:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333154432/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019426.468849:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734019426.468854:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734019426.468859:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333154432:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734019426.468868:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019423/real 1734019426] req@ffff8800a73cce00 x1818247333154560/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019426.468885:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333154560/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019426.468897:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734019426.468901:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734019426.468906:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333154560:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734019426.468915:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019423/real 1734019426] req@ffff8800a73cf480 x1818247333153792/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019426.468934:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333153792/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019426.468956:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734019426.468960:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734019426.468966:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333153792:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019426.468976:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019423/real 1734019426] req@ffff8800a73cd500 x1818247333153920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019426.468994:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333153920/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019426.469006:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734019426.469010:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734019426.469016:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333153920:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734019426.469026:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019423/real 1734019426] req@ffff8800a73cca80 x1818247333154048/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019426.469042:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333154048/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019426.469054:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734019426.469058:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734019426.469063:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333154048:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019426.554468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019427.556486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019428.470363:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019428.470375:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019428.470379:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019428.470390:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3691 00000100:00080000:3.0:1734019428.470399:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019428.470408:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019428.470445:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333154688/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019428.470479:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019428.470483:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019428.470485:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019428.470490:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3691 00000100:00080000:3.0:1734019428.470493:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019428.470497:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019428.470507:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019428.470518:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333154816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019428.470528:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019428.470531:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019428.470533:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019428.470536:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3691 00000100:00080000:3.0:1734019428.470539:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019428.470543:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019428.470552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019428.470560:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333154944/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019428.470568:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019428.470572:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019428.470573:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019428.470577:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3691 00000100:00080000:3.0:1734019428.470579:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019428.470583:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019428.470587:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333154688:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734019428.470587:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019428.470595:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333155072/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019428.470603:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019428.470607:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019428.470608:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019428.470611:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3691 00000100:00080000:3.0:1734019428.470614:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019428.470617:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019428.470621:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019428.470629:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333155200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019428.470637:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019428.470640:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019428.470642:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019428.470646:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3691 00000100:00080000:3.0:1734019428.470649:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019428.470652:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734019428.470653:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734019428.470657:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019428.470664:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333155328/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019428.470673:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734019428.470675:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019428/real 1734019428] req@ffff8800a73cca80 x1818247333154688/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019428.470676:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019428.470677:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019428.470681:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3691 00000100:00080000:3.0:1734019428.470683:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019428.470687:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019428.470691:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019428.470695:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333154688/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019428.470698:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333155456/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019428.470705:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019428.470743:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019428.470749:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333154688:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019428.470757:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333154816:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019428.470778:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019428.470787:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019428/real 1734019428] req@ffff8800a73cd500 x1818247333154816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019428.470799:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333154816/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019428.470807:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019428.470810:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019428.470813:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333154816:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019428.470819:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333154944:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019428.470833:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019428.470839:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019428/real 1734019428] req@ffff8800a73cf480 x1818247333154944/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019428.470850:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333154944/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019428.470858:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019428.470860:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019428.470863:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333154944:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019428.470886:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333155072:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019428.470899:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019428.470905:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019428/real 1734019428] req@ffff8800a73cce00 x1818247333155072/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019428.470915:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333155072/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019428.470922:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019428.470924:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019428.470927:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333155072:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019428.470931:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333155200:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019428.470943:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019428.470949:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019428/real 1734019428] req@ffff8800a73ced80 x1818247333155200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019428.470959:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333155200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019428.470973:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019428.470975:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019428.470979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333155200:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019428.470983:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333155328:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019428.470995:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019428.471001:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019428/real 1734019428] req@ffff8800a73cd880 x1818247333155328/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019428.471010:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333155328/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019428.471017:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019428.471019:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019428.471023:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333155328:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019428.471028:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333155456:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019428.471040:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019428.471045:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019428/real 1734019428] req@ffff8800a73cea00 x1818247333155456/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019428.471055:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333155456/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019428.471062:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019428.471064:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019428.471067:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333155456:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019428.558499:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019429.560472:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019430.562492:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019431.564505:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019432.566491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019433.478332:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019433.478342:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019433.478347:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019433.478357:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3696 00000100:00080000:3.0:1734019433.478367:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019433.478375:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019433.478412:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333155584/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019433.478451:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019433.478456:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019433.478458:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019433.478462:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3696 00000100:00080000:3.0:1734019433.478465:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019433.478469:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019433.478478:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019433.478491:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333155712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019433.478501:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019433.478504:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019433.478506:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019433.478509:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3696 00000100:00080000:3.0:1734019433.478512:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019433.478516:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019433.478520:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019433.478528:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333155840/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019433.478536:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019433.478539:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019433.478541:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019433.478544:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3696 00000100:00080000:3.0:1734019433.478547:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019433.478551:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019433.478554:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333155584:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734019433.478556:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019433.478563:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333155968/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019433.478572:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019433.478576:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019433.478577:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019433.478580:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3696 00000100:00080000:3.0:1734019433.478583:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019433.478586:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019433.478590:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019433.478598:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333156096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019433.478606:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019433.478609:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019433.478610:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019433.478614:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3696 00000100:00080000:3.0:1734019433.478617:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734019433.478619:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019433.478620:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019433.478650:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019433.478660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333156224/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019433.478666:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019433/real 1734019433] req@ffff8800a73ce680 x1818247333155584/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019433.478669:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019433.478672:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019433.478674:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019433.478678:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3696 00000100:00080000:3.0:1734019433.478681:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019433.478684:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019433.478686:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333155584/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734019433.478688:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1734019433.478697:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1734019433.478697:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333156352/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019433.478700:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019433.478705:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333155584:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019433.478712:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333155712:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019433.478732:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019433.478739:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019433/real 1734019433] req@ffff8800a73cea00 x1818247333155712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019433.478750:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333155712/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019433.478758:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019433.478761:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019433.478764:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333155712:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019433.478770:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333155840:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019433.478783:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019433.478789:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019433/real 1734019433] req@ffff8800a73cd880 x1818247333155840/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019433.478808:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333155840/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019433.478816:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019433.478818:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019433.478821:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333155840:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019433.478844:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333155968:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019433.478856:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019433.478863:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019433/real 1734019433] req@ffff8800a73ced80 x1818247333155968/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019433.478873:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333155968/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019433.478880:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019433.478882:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019433.478886:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333155968:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019433.478891:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333156096:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019433.478903:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019433.478909:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019433/real 1734019433] req@ffff8800a73cce00 x1818247333156096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019433.478918:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333156096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019433.478926:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019433.478929:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019433.478932:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333156096:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019433.478938:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333156224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019433.478950:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019433.478956:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019433/real 1734019433] req@ffff8800a73cf480 x1818247333156224/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019433.478966:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333156224/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019433.478974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019433.478976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019433.478979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333156224:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019433.478985:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333156352:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019433.478997:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019433.479002:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019433/real 1734019433] req@ffff8800a73cd500 x1818247333156352/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019433.479012:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333156352/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019433.479019:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019433.479021:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019433.479024:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333156352:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019433.568479:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019434.570487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019435.572465:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019436.574477:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019437.576440:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019438.486368:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019438.486378:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019438.486383:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019438.486394:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3701 00000100:00080000:3.0:1734019438.486404:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019438.486412:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019438.486448:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333156480/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019438.486483:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019438.486487:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019438.486489:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019438.486494:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3701 00000100:00080000:3.0:1734019438.486498:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019438.486502:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019438.486512:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019438.486522:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333156608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019438.486533:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019438.486536:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019438.486575:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019438.486580:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3701 00000100:00080000:3.0:1734019438.486584:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019438.486588:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019438.486593:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019438.486603:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333156736/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019438.486613:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734019438.486615:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333156480:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019438.486616:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019438.486618:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019438.486621:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3701 00000100:00080000:3.0:1734019438.486624:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019438.486627:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019438.486632:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019438.486639:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333156864/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019438.486653:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019438.486657:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019438.486658:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019438.486662:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3701 00000100:00080000:3.0:1734019438.486664:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019438.486668:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019438.486672:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019438.486680:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333156992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734019438.486685:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019438.486688:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019438.486691:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019438.486692:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019438.486696:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3701 00000100:00080000:3.0:1734019438.486701:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019438.486704:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019438.486707:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019438/real 1734019438] req@ffff8800a73cca80 x1818247333156480/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734019438.486708:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019438.486716:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333157120/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019438.486724:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734019438.486726:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333156480/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019438.486727:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019438.486728:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019438.486732:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3701 00000100:00080000:3.0:1734019438.486735:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1734019438.486736:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734019438.486738:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:0.0:1734019438.486740:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000008:00100000:3.0:1734019438.486742:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019438.486744:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333156480:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019438.486752:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333156608:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734019438.486752:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333157248/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734019438.486768:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019438.486775:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019438/real 1734019438] req@ffff8800a73cc380 x1818247333156608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019438.486786:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333156608/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019438.486794:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019438.486796:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019438.486800:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333156608:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019438.486822:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333156736:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019438.486836:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019438.486842:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019438/real 1734019438] req@ffff8800a73cf100 x1818247333156736/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019438.486852:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333156736/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019438.486860:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019438.486863:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019438.486866:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333156736:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019438.486871:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333156864:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019438.486883:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019438.486888:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019438/real 1734019438] req@ffff8800a73cc700 x1818247333156864/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019438.486898:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333156864/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019438.486905:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019438.486907:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019438.486910:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333156864:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019438.486916:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333156992:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019438.486929:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019438.486934:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019438/real 1734019438] req@ffff8800a73cdf80 x1818247333156992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019438.486944:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333156992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019438.486951:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019438.486953:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019438.486956:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333156992:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019438.486962:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333157120:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019438.486974:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019438.486980:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019438/real 1734019438] req@ffff8800a73cf800 x1818247333157120/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019438.486990:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333157120/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019438.486998:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019438.487000:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019438.487003:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333157120:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019438.487008:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333157248:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019438.487020:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019438.487026:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019438/real 1734019438] req@ffff8800a73cfb80 x1818247333157248/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019438.487035:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333157248/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019438.487042:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019438.487044:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019438.487047:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333157248:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019438.578472:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019439.580439:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019440.582475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019441.584643:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019442.586536:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019443.494341:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019443.494354:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019443.494360:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019443.494374:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3706 00000100:00080000:3.0:1734019443.494385:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019443.494395:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019443.494443:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333157376/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019443.494514:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019443.494519:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019443.494521:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019443.494526:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3706 00000100:00080000:3.0:1734019443.494530:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019443.494535:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019443.494547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019443.494564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333157504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019443.494575:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019443.494579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019443.494581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019443.494584:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3706 00000100:00080000:3.0:1734019443.494587:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019443.494591:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019443.494595:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019443.494604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333157632/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019443.494613:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019443.494617:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019443.494618:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019443.494622:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3706 00000100:00080000:3.0:1734019443.494624:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734019443.494628:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333157376:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019443.494628:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019443.494633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019443.494641:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333157760/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019443.494650:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019443.494653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019443.494655:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019443.494660:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3706 00000100:00080000:3.0:1734019443.494662:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019443.494666:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019443.494670:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019443.494679:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333157888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019443.494687:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019443.494690:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019443.494692:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019443.494696:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3706 00000100:00080000:3.0:1734019443.494700:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019443.494703:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019443.494708:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734019443.494713:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019443.494716:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333158016/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019443.494724:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019443.494727:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019443.494729:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019443.494732:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3706 00000100:00080000:3.0:1734019443.494735:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019443.494738:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019443.494739:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019443/real 1734019443] req@ffff8800a73cdc00 x1818247333157376/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734019443.494743:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019443.494752:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333158144/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019443.494762:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333157376/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019443.494774:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019443.494778:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019443.494782:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333157376:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019443.494790:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333157504:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019443.494809:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019443.494816:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019443/real 1734019443] req@ffff8800a73cfb80 x1818247333157504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019443.494828:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333157504/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019443.494836:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019443.494838:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019443.494842:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333157504:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019443.494870:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333157632:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019443.494884:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019443.494891:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019443/real 1734019443] req@ffff8800a73cf800 x1818247333157632/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019443.494901:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333157632/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019443.494909:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019443.494913:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019443.494916:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333157632:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019443.494922:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333157760:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019443.494934:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019443.494940:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019443/real 1734019443] req@ffff8800a73cdf80 x1818247333157760/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019443.494950:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333157760/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019443.494958:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019443.494961:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019443.494964:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333157760:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019443.494969:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333157888:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019443.494982:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019443.494988:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019443/real 1734019443] req@ffff8800a73cc700 x1818247333157888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019443.494997:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333157888/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019443.495005:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019443.495007:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019443.495011:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333157888:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019443.495017:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333158016:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019443.495033:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019443.495039:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019443/real 1734019443] req@ffff8800a73cf100 x1818247333158016/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019443.495050:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333158016/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019443.495057:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019443.495059:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019443.495062:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333158016:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019443.495067:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333158144:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019443.495079:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019443.495085:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019443/real 1734019443] req@ffff8800a73cc380 x1818247333158144/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019443.495095:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333158144/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019443.495102:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019443.495104:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019443.495107:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333158144:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019443.588487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019444.590499:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019445.592507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019446.594375:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019447.596472:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019448.502342:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019448.502353:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019448.502359:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019448.502399:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3711 00000100:00080000:3.0:1734019448.502410:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019448.502420:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019448.502470:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333158272/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019448.502507:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019448.502512:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019448.502514:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019448.502518:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3711 00000100:00080000:3.0:1734019448.502522:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019448.502526:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019448.502538:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019448.502550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333158400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019448.502560:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019448.502564:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019448.502567:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019448.502571:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3711 00000100:00080000:3.0:1734019448.502574:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019448.502578:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019448.502583:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019448.502591:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333158528/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019448.502601:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019448.502604:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019448.502605:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019448.502609:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3711 00000100:00080000:3.0:1734019448.502612:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734019448.502613:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333158272:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019448.502615:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019448.502620:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019448.502628:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333158656/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019448.502636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019448.502639:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019448.502641:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019448.502644:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3711 00000100:00080000:3.0:1734019448.502647:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019448.502651:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019448.502656:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019448.502665:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333158784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019448.502674:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019448.502678:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019448.502680:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019448.502684:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3711 00000100:00080000:3.0:1734019448.502687:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019448.502691:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019448.502695:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734019448.502700:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019448.502703:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333158912/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019448.502711:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019448.502714:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019448.502716:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019448.502719:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3711 00000100:00080000:3.0:1734019448.502722:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019448.502725:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019448.502728:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019448/real 1734019448] req@ffff8800a73cca80 x1818247333158272/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734019448.502730:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019448.502740:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333159040/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019448.502758:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333158272/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019448.502769:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019448.502773:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019448.502777:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333158272:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019448.502785:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333158400:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019448.502804:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019448.502810:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019448/real 1734019448] req@ffff8800a73cd500 x1818247333158400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019448.502822:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333158400/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019448.502830:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019448.502832:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019448.502835:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333158400:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019448.502861:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333158528:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019448.502876:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019448.502882:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019448/real 1734019448] req@ffff8800a73cf480 x1818247333158528/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019448.502892:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333158528/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019448.502901:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019448.502904:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019448.502907:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333158528:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019448.502913:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333158656:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019448.502925:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019448.502931:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019448/real 1734019448] req@ffff8800a73cce00 x1818247333158656/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019448.502941:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333158656/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019448.502949:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019448.502951:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019448.502954:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333158656:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019448.502959:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333158784:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019448.502972:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019448.502977:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019448/real 1734019448] req@ffff8800a73ced80 x1818247333158784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019448.502987:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333158784/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019448.502994:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019448.502997:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019448.502999:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333158784:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019448.503004:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333158912:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019448.503021:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019448.503027:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019448/real 1734019448] req@ffff8800a73cd880 x1818247333158912/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019448.503038:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333158912/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019448.503045:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019448.503047:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019448.503050:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333158912:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019448.503055:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333159040:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019448.503068:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019448.503073:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019448/real 1734019448] req@ffff8800a73cea00 x1818247333159040/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019448.503083:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333159040/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019448.503090:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019448.503092:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019448.503095:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333159040:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019448.598470:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019449.600451:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019450.602478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019451.604579:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019452.606490:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019453.510373:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019453.510383:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019453.510387:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019453.510397:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3716 00000100:00080000:3.0:1734019453.510407:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019453.510415:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019453.510450:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333159168/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019453.510484:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019453.510490:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019453.510493:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019453.510500:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3716 00000100:00080000:3.0:1734019453.510504:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019453.510511:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019453.510524:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019453.510545:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333159296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019453.510552:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333159168:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019453.510561:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019453.510565:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019453.510568:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019453.510573:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3716 00000100:00080000:3.0:1734019453.510576:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019453.510581:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019453.510587:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019453.510600:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333159424/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019453.510613:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019453.510617:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019453.510620:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1734019453.510623:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019453.510625:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3716 00000100:00080000:3.0:1734019453.510628:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019453.510633:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019453.510640:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019453.510647:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019453/real 1734019453] req@ffff8800a73ce680 x1818247333159168/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019453.510652:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333159552/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019453.510666:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019453.510671:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019453.510673:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019453.510678:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3716 00000100:00100000:0.0:1734019453.510681:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333159168/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019453.510682:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019453.510687:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019453.510693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1734019453.510696:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019453.510701:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734019453.510706:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333159680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019453.510707:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333159168:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019453.510718:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019453.510722:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019453.510725:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019453.510730:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3716 00000100:00100000:0.0:1734019453.510733:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333159296:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734019453.510735:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019453.510740:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019453.510747:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734019453.510759:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019453.510760:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333159808/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019453.510771:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019453/real 1734019453] req@ffff8800a73cea00 x1818247333159296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019453.510774:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019453.510780:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019453.510782:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019453.510787:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3716 00000100:00100000:0.0:1734019453.510788:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333159296/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019453.510792:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019453.510797:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:0.0:1734019453.510801:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1734019453.510803:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1734019453.510804:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019453.510810:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333159296:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734019453.510815:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333159936/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019453.510819:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333159424:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019453.510841:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019453.510852:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019453/real 1734019453] req@ffff8800a73cd880 x1818247333159424/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019453.510869:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333159424/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019453.510881:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019453.510886:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019453.510891:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333159424:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019453.510900:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333159552:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019453.510926:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019453.510937:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019453/real 1734019453] req@ffff8800a73ced80 x1818247333159552/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019453.510949:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333159552/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019453.510956:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019453.510958:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019453.510962:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333159552:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019453.510967:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333159680:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019453.510984:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019453.510990:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019453/real 1734019453] req@ffff8800a73cce00 x1818247333159680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019453.511001:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333159680/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019453.511008:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019453.511010:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019453.511013:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333159680:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019453.511031:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333159808:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019453.511047:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019453.511053:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019453/real 1734019453] req@ffff8800a73cf480 x1818247333159808/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019453.511064:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333159808/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019453.511071:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019453.511073:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019453.511076:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333159808:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019453.511080:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333159936:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019453.511092:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019453.511097:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019453/real 1734019453] req@ffff8800a73cd500 x1818247333159936/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019453.511107:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333159936/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019453.511114:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019453.511116:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019453.511119:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333159936:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019453.608438:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019454.610440:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019455.612494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019456.614471:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019457.616488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019458.518352:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019458.518362:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019458.518366:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019458.518377:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3721 00000100:00080000:3.0:1734019458.518386:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019458.518394:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019458.518431:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333160064/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019458.518467:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019458.518473:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019458.518476:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019458.518483:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3721 00000100:00080000:3.0:1734019458.518488:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019458.518494:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019458.518507:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019458.518525:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333160192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019458.518535:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333160064:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019458.518541:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019458.518546:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019458.518548:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019458.518554:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3721 00000100:00080000:3.0:1734019458.518557:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019458.518563:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019458.518569:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019458.518581:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333160320/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019458.518593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019458.518597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019458.518600:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1734019458.518605:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019458.518606:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3721 00000100:00080000:3.0:1734019458.518610:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019458.518614:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019458.518620:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019458.518628:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019458/real 1734019458] req@ffff8800a73cca80 x1818247333160064/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019458.518632:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333160448/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019458.518645:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019458.518651:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019458.518653:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019458.518658:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3721 00000100:00100000:0.0:1734019458.518662:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333160064/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019458.518662:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019458.518667:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019458.518674:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1734019458.518676:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019458.518681:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734019458.518686:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333160576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019458.518687:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333160064:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019458.518700:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019458.518704:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019458.518706:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019458.518712:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3721 00000100:00100000:0.0:1734019458.518713:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333160192:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734019458.518716:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019458.518721:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019458.518727:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734019458.518740:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019458.518741:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333160704/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019458.518752:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019458/real 1734019458] req@ffff8800a73cc380 x1818247333160192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019458.518756:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019458.518761:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019458.518764:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019458.518769:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3721 00000100:00100000:0.0:1734019458.518770:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333160192/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019458.518773:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019458.518778:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:0.0:1734019458.518782:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1734019458.518784:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1734019458.518786:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019458.518791:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333160192:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734019458.518796:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333160832/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019458.518801:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333160320:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019458.518823:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019458.518833:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019458/real 1734019458] req@ffff8800a73cf100 x1818247333160320/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019458.518850:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333160320/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019458.518863:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019458.518868:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019458.518873:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333160320:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019458.518881:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333160448:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019458.518908:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019458.518919:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019458/real 1734019458] req@ffff8800a73cc700 x1818247333160448/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019458.518932:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333160448/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019458.518940:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019458.518942:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019458.518945:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333160448:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019458.518951:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333160576:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019458.518966:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019458.518973:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019458/real 1734019458] req@ffff8800a73cdf80 x1818247333160576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019458.518983:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333160576/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019458.518991:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019458.518993:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019458.518996:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333160576:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019458.519013:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333160704:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019458.519029:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019458.519035:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019458/real 1734019458] req@ffff8800a73cf800 x1818247333160704/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019458.519046:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333160704/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019458.519053:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019458.519055:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019458.519058:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333160704:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019458.519063:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333160832:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019458.519074:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019458.519080:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019458/real 1734019458] req@ffff8800a73cfb80 x1818247333160832/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019458.519090:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333160832/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019458.519097:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019458.519099:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019458.519102:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333160832:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019458.618411:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019459.620468:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019460.622494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019461.624491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019462.626513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019463.526343:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019463.526352:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019463.526357:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019463.526368:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3726 00000100:00080000:3.0:1734019463.526378:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019463.526385:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019463.526423:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333160960/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019463.526457:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019463.526461:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019463.526463:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019463.526468:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3726 00000100:00080000:3.0:1734019463.526471:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019463.526475:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019463.526485:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019463.526498:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333161088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019463.526508:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019463.526511:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019463.526513:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019463.526517:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3726 00000100:00080000:3.0:1734019463.526520:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019463.526523:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019463.526528:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019463.526536:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333161216/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019463.526544:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019463.526547:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019463.526549:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019463.526552:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3726 00000100:00080000:3.0:1734019463.526554:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019463.526558:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019463.526562:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019463.526566:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333160960:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019463.526569:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333161344/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019463.526579:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019463.526582:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019463.526584:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019463.526587:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3726 00000100:00080000:3.0:1734019463.526589:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019463.526593:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019463.526597:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019463.526604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333161472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019463.526612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019463.526615:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019463.526617:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019463.526621:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3726 00000100:00080000:3.0:1734019463.526623:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019463.526626:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019463.526631:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734019463.526632:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019463.526638:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333161600/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019463.526646:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019463.526649:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019463.526651:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734019463.526653:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019463/real 1734019463] req@ffff8800a73cdc00 x1818247333160960/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019463.526654:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3726 00000100:00080000:3.0:1734019463.526658:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019463.526661:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019463.526666:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019463.526673:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333161728/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019463.526681:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333160960/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019463.526692:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019463.526695:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019463.526699:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333160960:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019463.526706:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333161088:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019463.526724:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019463.526730:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019463/real 1734019463] req@ffff8800a73cfb80 x1818247333161088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019463.526741:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333161088/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019463.526749:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019463.526752:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019463.526755:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333161088:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019463.526761:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333161216:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019463.526774:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019463.526780:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019463/real 1734019463] req@ffff8800a73cf800 x1818247333161216/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019463.526790:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333161216/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019463.526798:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019463.526801:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019463.526804:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333161216:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019463.526828:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333161344:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019463.526841:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019463.526847:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019463/real 1734019463] req@ffff8800a73cdf80 x1818247333161344/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019463.526857:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333161344/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019463.526865:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019463.526867:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019463.526870:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333161344:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019463.526875:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333161472:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019463.526886:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019463.526892:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019463/real 1734019463] req@ffff8800a73cc700 x1818247333161472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019463.526901:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333161472/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019463.526909:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019463.526911:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019463.526914:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333161472:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019463.526919:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333161600:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019463.526934:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019463.526940:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019463/real 1734019463] req@ffff8800a73cf100 x1818247333161600/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019463.526950:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333161600/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019463.526958:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019463.526960:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019463.526963:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333161600:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019463.526968:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333161728:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019463.526979:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019463.526985:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019463/real 1734019463] req@ffff8800a73cc380 x1818247333161728/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019463.526994:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333161728/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019463.527002:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019463.527004:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019463.527007:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333161728:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019463.628490:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019464.630474:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019465.632470:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019466.634512:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019467.636458:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019468.534350:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019468.534359:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019468.534364:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019468.534374:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3731 00000100:00080000:3.0:1734019468.534383:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019468.534391:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019468.534427:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333161856/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019468.534459:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019468.534464:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019468.534466:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019468.534470:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3731 00000100:00080000:3.0:1734019468.534474:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019468.534478:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019468.534487:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019468.534499:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333161984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019468.534509:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019468.534513:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019468.534514:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019468.534518:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3731 00000100:00080000:3.0:1734019468.534521:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019468.534524:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019468.534529:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019468.534537:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333162112/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019468.534545:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019468.534548:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019468.534550:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019468.534553:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3731 00000100:00080000:3.0:1734019468.534556:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019468.534559:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019468.534562:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333161856:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734019468.534563:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019468.534570:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333162240/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019468.534579:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019468.534583:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019468.534584:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019468.534588:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3731 00000100:00080000:3.0:1734019468.534591:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019468.534594:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019468.534598:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019468.534605:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333162368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019468.534613:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019468.534616:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019468.534618:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019468.534622:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3731 00000100:00080000:3.0:1734019468.534624:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734019468.534628:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019468.534628:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019468.534632:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019468.534639:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333162496/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019468.534647:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734019468.534650:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019468/real 1734019468] req@ffff8800a73cca80 x1818247333161856/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019468.534650:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019468.534652:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019468.534656:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3731 00000100:00080000:3.0:1734019468.534658:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019468.534662:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019468.534666:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019468.534674:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333162624/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019468.534677:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333161856/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019468.534688:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019468.534691:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019468.534695:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333161856:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019468.534702:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333161984:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019468.534719:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019468.534726:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019468/real 1734019468] req@ffff8800a73cd500 x1818247333161984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019468.534737:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333161984/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019468.534745:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019468.534747:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019468.534751:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333161984:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019468.534756:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333162112:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019468.534769:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019468.534775:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019468/real 1734019468] req@ffff8800a73cf480 x1818247333162112/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019468.534785:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333162112/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019468.534792:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019468.534795:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019468.534798:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333162112:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019468.534823:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333162240:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019468.534836:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019468.534842:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019468/real 1734019468] req@ffff8800a73cce00 x1818247333162240/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019468.534853:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333162240/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019468.534860:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019468.534862:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019468.534865:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333162240:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019468.534870:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333162368:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019468.534881:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019468.534887:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019468/real 1734019468] req@ffff8800a73ced80 x1818247333162368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019468.534897:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333162368/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019468.534904:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019468.534906:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019468.534909:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333162368:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019468.534914:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333162496:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019468.534929:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019468.534935:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019468/real 1734019468] req@ffff8800a73cd880 x1818247333162496/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019468.534946:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333162496/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019468.534953:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019468.534955:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019468.534957:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333162496:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019468.534962:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333162624:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019468.534974:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019468.534980:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019468/real 1734019468] req@ffff8800a73cea00 x1818247333162624/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019468.534990:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333162624/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019468.534998:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019468.535000:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019468.535002:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333162624:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019468.638496:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019469.640478:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019470.642487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019471.644499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019472.646466:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019473.542332:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019473.542341:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019473.542346:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019473.542356:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3736 00000100:00080000:3.0:1734019473.542366:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019473.542374:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019473.542408:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333162752/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019473.542443:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019473.542447:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019473.542449:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019473.542453:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3736 00000100:00080000:3.0:1734019473.542457:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019473.542461:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019473.542470:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019473.542482:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333162880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019473.542492:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019473.542495:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019473.542498:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019473.542501:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3736 00000100:00080000:3.0:1734019473.542504:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019473.542507:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019473.542512:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019473.542521:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333163008/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019473.542530:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019473.542533:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019473.542534:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019473.542537:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3736 00000100:00080000:3.0:1734019473.542540:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019473.542543:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019473.542547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019473.542554:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333163136/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019473.542564:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019473.542567:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019473.542569:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734019473.542573:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333162752:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019473.542573:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3736 00000100:00080000:3.0:1734019473.542575:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019473.542579:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019473.542583:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019473.542591:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333163264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019473.542601:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019473.542604:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019473.542605:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019473.542609:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3736 00000100:00080000:3.0:1734019473.542612:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019473.542615:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019473.542619:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019473.542627:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333163392/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019473.542635:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019473.542638:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019473.542639:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019473.542642:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3736 00000100:00080000:3.0:1734019473.542646:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019473.542649:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019473.542653:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019473.542661:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333163520/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734019473.542666:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019473.542693:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019473/real 1734019473] req@ffff8800a73ce680 x1818247333162752/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019473.542724:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333162752/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019473.542737:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019473.542741:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019473.542746:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333162752:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019473.542754:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333162880:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019473.542774:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019473.542780:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019473/real 1734019473] req@ffff8800a73cea00 x1818247333162880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019473.542792:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333162880/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019473.542800:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019473.542803:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019473.542806:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333162880:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019473.542812:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333163008:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019473.542827:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019473.542833:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019473/real 1734019473] req@ffff8800a73cd880 x1818247333163008/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019473.542843:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333163008/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019473.542850:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019473.542854:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019473.542857:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333163008:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019473.542887:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333163136:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019473.542900:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019473.542907:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019473/real 1734019473] req@ffff8800a73ced80 x1818247333163136/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019473.542917:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333163136/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019473.542925:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019473.542927:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019473.542930:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333163136:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019473.542936:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333163264:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019473.542978:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019473.542987:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019473/real 1734019473] req@ffff8800a73cce00 x1818247333163264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019473.542997:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333163264/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019473.543005:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019473.543008:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019473.543011:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333163264:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019473.543017:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333163392:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019473.543037:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019473.543043:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019473/real 1734019473] req@ffff8800a73cf480 x1818247333163392/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019473.543054:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333163392/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019473.543061:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019473.543063:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019473.543066:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333163392:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019473.543072:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333163520:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019473.543085:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019473.543091:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019473/real 1734019473] req@ffff8800a73cd500 x1818247333163520/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019473.543100:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333163520/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019473.543108:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019473.543110:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019473.543113:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333163520:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019473.648502:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019474.650487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019475.652501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019476.654513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019477.656460:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019478.550358:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019478.550367:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019478.550372:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019478.550383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3741 00000100:00080000:3.0:1734019478.550393:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019478.550401:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019478.550438:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333163648/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019478.550471:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019478.550475:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019478.550477:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019478.550482:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3741 00000100:00080000:3.0:1734019478.550485:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019478.550489:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019478.550499:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019478.550510:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333163776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019478.550520:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019478.550523:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019478.550525:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019478.550529:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3741 00000100:00080000:3.0:1734019478.550532:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019478.550535:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019478.550540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019478.550548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333163904/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019478.550557:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019478.550560:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019478.550562:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019478.550565:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3741 00000100:00080000:3.0:1734019478.550568:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019478.550571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019478.550576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019478.550579:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333163648:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019478.550583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333164032/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019478.550592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019478.550595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019478.550597:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019478.550600:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3741 00000100:00080000:3.0:1734019478.550603:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019478.550606:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019478.550610:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019478.550618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333164160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019478.550627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019478.550630:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019478.550631:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019478.550635:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3741 00000100:00080000:3.0:1734019478.550638:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019478.550642:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019478.550646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734019478.550647:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019478.550654:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333164288/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019478.550661:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019478.550665:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019478.550666:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734019478.550668:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019478/real 1734019478] req@ffff8800a73cca80 x1818247333163648/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019478.550669:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3741 00000100:00080000:3.0:1734019478.550672:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019478.550675:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019478.550679:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019478.550688:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333164416/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019478.550696:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333163648/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019478.550706:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019478.550709:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019478.550714:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333163648:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019478.550721:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333163776:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019478.550738:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019478.550745:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019478/real 1734019478] req@ffff8800a73cc380 x1818247333163776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019478.550756:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333163776/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019478.550764:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019478.550766:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019478.550770:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333163776:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019478.550776:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333163904:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019478.550789:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019478.550794:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019478/real 1734019478] req@ffff8800a73cf100 x1818247333163904/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019478.550804:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333163904/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019478.550812:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019478.550815:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019478.550818:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333163904:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019478.550843:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333164032:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019478.550855:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019478.550861:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019478/real 1734019478] req@ffff8800a73cc700 x1818247333164032/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019478.550910:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333164032/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019478.550919:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019478.550921:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019478.550924:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333164032:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019478.550930:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333164160:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019478.550945:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019478.550952:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019478/real 1734019478] req@ffff8800a73cdf80 x1818247333164160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019478.550962:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333164160/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019478.550970:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019478.550972:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019478.550975:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333164160:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019478.550980:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333164288:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019478.550996:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019478.551002:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019478/real 1734019478] req@ffff8800a73cf800 x1818247333164288/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019478.551012:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333164288/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019478.551019:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019478.551022:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019478.551024:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333164288:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019478.551029:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333164416:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019478.551042:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019478.551047:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019478/real 1734019478] req@ffff8800a73cfb80 x1818247333164416/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019478.551057:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333164416/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019478.551064:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019478.551066:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019478.551069:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333164416:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019478.658491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019479.660486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019480.662445:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019481.664441:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019482.666475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019483.558355:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019483.558364:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019483.558368:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019483.558379:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3746 00000100:00080000:3.0:1734019483.558388:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019483.558396:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019483.558432:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333164544/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019483.558465:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019483.558469:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019483.558471:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019483.558476:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3746 00000100:00080000:3.0:1734019483.558479:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019483.558483:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019483.558494:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019483.558507:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333164672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019483.558517:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019483.558520:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019483.558522:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019483.558526:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3746 00000100:00080000:3.0:1734019483.558528:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019483.558531:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019483.558536:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019483.558544:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333164800/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019483.558553:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019483.558556:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019483.558557:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019483.558561:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3746 00000100:00080000:3.0:1734019483.558564:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019483.558567:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019483.558571:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333164544:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734019483.558571:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019483.558578:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333164928/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019483.558587:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019483.558590:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019483.558592:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019483.558595:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3746 00000100:00080000:3.0:1734019483.558598:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019483.558601:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019483.558605:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019483.558613:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333165056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019483.558621:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019483.558624:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019483.558625:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019483.558629:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3746 00000100:00080000:3.0:1734019483.558632:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019483.558635:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734019483.558636:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734019483.558640:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019483.558647:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333165184/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019483.558656:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734019483.558658:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019483/real 1734019483] req@ffff8800a73cdc00 x1818247333164544/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019483.558659:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019483.558660:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019483.558664:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3746 00000100:00080000:3.0:1734019483.558666:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019483.558670:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019483.558674:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019483.558681:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333165312/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019483.558685:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333164544/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019483.558695:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019483.558698:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019483.558703:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333164544:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019483.558710:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333164672:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019483.558727:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019483.558734:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019483/real 1734019483] req@ffff8800a73cfb80 x1818247333164672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019483.558745:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333164672/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019483.558753:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019483.558755:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019483.558758:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333164672:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019483.558764:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333164800:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019483.558777:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019483.558819:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019483/real 1734019483] req@ffff8800a73cf800 x1818247333164800/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019483.558830:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333164800/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019483.558838:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019483.558841:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019483.558844:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333164800:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019483.558868:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333164928:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019483.558884:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019483.558891:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019483/real 1734019483] req@ffff8800a73cdf80 x1818247333164928/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019483.558901:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333164928/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019483.558909:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019483.558911:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019483.558914:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333164928:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019483.558919:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333165056:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019483.558931:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019483.558937:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019483/real 1734019483] req@ffff8800a73cc700 x1818247333165056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019483.558946:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333165056/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019483.558954:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019483.558956:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019483.558959:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333165056:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019483.558964:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333165184:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019483.558979:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019483.558985:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019483/real 1734019483] req@ffff8800a73cf100 x1818247333165184/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019483.558995:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333165184/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019483.559002:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019483.559004:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019483.559007:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333165184:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019483.559012:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333165312:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019483.559024:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019483.559029:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019483/real 1734019483] req@ffff8800a73cc380 x1818247333165312/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019483.559048:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333165312/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019483.559056:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019483.559058:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019483.559061:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333165312:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019483.668449:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019484.670519:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019485.672423:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019486.674508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019487.676503:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019488.566365:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019488.566376:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019488.566382:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019488.566396:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3751 00000100:00080000:3.0:1734019488.566409:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019488.566419:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019488.566469:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333165440/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019488.566506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019488.566510:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019488.566513:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019488.566518:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3751 00000100:00080000:3.0:1734019488.566522:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019488.566526:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019488.566538:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019488.566549:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333165568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019488.566559:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019488.566564:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019488.566566:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019488.566570:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3751 00000100:00080000:3.0:1734019488.566573:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019488.566577:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019488.566581:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019488.566590:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333165696/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019488.566599:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019488.566603:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019488.566604:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019488.566608:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3751 00000100:00080000:3.0:1734019488.566611:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734019488.566613:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333165440:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019488.566614:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019488.566619:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019488.566627:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333165824/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019488.566636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019488.566640:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019488.566641:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019488.566645:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3751 00000100:00080000:3.0:1734019488.566648:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019488.566651:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019488.566656:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019488.566664:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333165952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019488.566672:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019488.566675:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019488.566677:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019488.566681:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3751 00000100:00080000:3.0:1734019488.566685:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019488.566688:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019488.566693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019488.566731:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333166080/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019488.566741:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019488.566744:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019488.566746:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734019488.566749:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333165568:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734019488.566750:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3751 00000100:00080000:3.0:1734019488.566753:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019488.566757:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019488.566762:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019488.566773:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333166208/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019488.566785:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333165696:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019488.566803:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333165824:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019488.566821:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333165952:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019488.566836:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333166080:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019488.566853:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333166208:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019488.678487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019489.680473:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019490.682423:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1734019491.572423:0:1956:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:1.0:1734019491.572457:0:1956:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1734019491.572471:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019491.572495:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019491.572500:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019491.572503:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019491.572508:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019491.572511:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019491.572515:0:1956:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:2.0:1734019491.572574:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019488/real 1734019491] req@ffff8800a73cf480 x1818247333165696/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019491.572600:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333165696/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019491.572609:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019491.572611:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019491.572615:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333165696:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019491.572621:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019488/real 1734019491] req@ffff8800a73cce00 x1818247333165824/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019491.572629:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333165824/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019491.572633:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019491.572635:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019491.572637:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333165824:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019491.572641:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019488/real 1734019491] req@ffff8800a73ced80 x1818247333165952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019491.572669:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333165952/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019491.572674:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019491.572675:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019491.572677:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333165952:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019491.572681:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019488/real 1734019491] req@ffff8800a73cd880 x1818247333166080/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019491.572688:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333166080/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019491.572692:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019491.572693:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019491.572695:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333166080:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019491.572701:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019488/real 1734019491] req@ffff8800a73cea00 x1818247333166208/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019491.572707:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333166208/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019491.572711:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019491.572713:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019491.572714:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333166208:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019491.572718:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019488/real 1734019491] req@ffff8800a73cca80 x1818247333165440/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019491.572725:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333165440/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019491.572729:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019491.572730:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019491.572732:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333165440:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019491.572736:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019488/real 1734019491] req@ffff8800a73cd500 x1818247333165568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019491.572742:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333165568/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019491.572745:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019491.572747:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019491.572749:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333165568:192.168.201.152@tcp:38:kworker.0 00010000:00010000:3.0:1734019491.684445:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019492.686475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019493.574351:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019493.574360:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019493.574365:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019493.574376:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3756 00000100:00080000:3.0:1734019493.574387:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019493.574395:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019493.574431:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333166336/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019493.574465:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019493.574470:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019493.574472:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019493.574476:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3756 00000100:00080000:3.0:1734019493.574480:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019493.574484:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019493.574493:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019493.574507:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333166464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019493.574517:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019493.574520:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019493.574522:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019493.574526:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3756 00000100:00080000:3.0:1734019493.574528:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019493.574531:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019493.574536:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019493.574545:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333166592/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019493.574553:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019493.574556:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019493.574558:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019493.574562:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3756 00000100:00080000:3.0:1734019493.574564:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019493.574568:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019493.574572:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019493.574580:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333166720/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019493.574590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019493.574593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019493.574594:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019493.574599:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3756 00000100:00100000:2.0:1734019493.574600:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333166336:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019493.574601:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019493.574605:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019493.574610:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019493.574665:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333166848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019493.574671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019493.574673:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019493.574674:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019493.574677:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3756 00000100:00080000:3.0:1734019493.574679:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019493.574682:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019493.574685:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019493.574690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333166976/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019493.574694:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019493.574696:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019493.574697:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019493.574700:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3756 00000100:00080000:3.0:1734019493.574734:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019493.574738:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019493.574743:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019493.574746:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019493.574753:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333167104/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019493.574775:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019493/real 1734019493] req@ffff8800a73ce680 x1818247333166336/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019493.574798:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333166336/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019493.574811:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019493.574815:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019493.574820:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333166336:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019493.574829:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333166464:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019493.574849:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019493.574856:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019493/real 1734019493] req@ffff8800a73cd500 x1818247333166464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019493.574867:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333166464/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019493.574875:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019493.574878:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019493.574881:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333166464:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019493.574888:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333166592:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019493.574903:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019493.574909:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019493/real 1734019493] req@ffff8800a73cca80 x1818247333166592/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019493.574919:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333166592/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019493.574928:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019493.574931:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019493.574934:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333166592:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019493.574963:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333166720:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019493.574976:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019493.574982:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019493/real 1734019493] req@ffff8800a73cea00 x1818247333166720/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019493.574992:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333166720/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019493.575001:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019493.575003:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019493.575006:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333166720:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019493.575012:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333166848:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019493.575024:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019493.575030:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019493/real 1734019493] req@ffff8800a73cd880 x1818247333166848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019493.575040:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333166848/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019493.575048:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019493.575050:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019493.575053:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333166848:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019493.575060:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333166976:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019493.575073:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019493.575079:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019493/real 1734019493] req@ffff8800a73ced80 x1818247333166976/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019493.575090:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333166976/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019493.575097:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019493.575099:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019493.575102:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333166976:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019493.575108:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333167104:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019493.575121:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019493.575127:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019493/real 1734019493] req@ffff8800a73cce00 x1818247333167104/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019493.575136:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333167104/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019493.575143:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019493.575146:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019493.575149:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333167104:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019493.688475:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019494.690488:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019495.692508:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019496.694466:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019497.696411:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019498.582371:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019498.582384:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019498.582390:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019498.582404:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3761 00000100:00080000:3.0:1734019498.582415:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019498.582425:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019498.582473:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333167232/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019498.582510:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019498.582515:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019498.582517:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019498.582522:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3761 00000100:00080000:3.0:1734019498.582525:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019498.582556:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019498.582568:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019498.582582:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333167360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019498.582593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019498.582597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019498.582599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019498.582603:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3761 00000100:00080000:3.0:1734019498.582607:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019498.582610:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019498.582616:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:2.0:1734019498.582621:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333167232:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019498.582625:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333167488/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019498.582634:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019498.582637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019498.582639:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019498.582643:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3761 00000100:00080000:3.0:1734019498.582646:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019498.582650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019498.582655:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019498.582663:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333167616/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019498.582671:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019498.582674:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019498.582676:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019498.582679:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3761 00000100:00080000:3.0:1734019498.582694:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019498.582698:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019498.582703:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1734019498.582710:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019498.582712:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333167744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019498.582721:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019498.582725:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019498.582726:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019498.582731:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3761 00000100:00080000:3.0:1734019498.582734:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734019498.582736:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019498/real 1734019498] req@ffff8800a73cf480 x1818247333167232/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019498.582738:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019498.582742:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019498.582749:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333167872/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019498.582757:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734019498.582759:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333167232/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019498.582761:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019498.582762:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019498.582766:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3761 00000100:00080000:3.0:1734019498.582769:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1734019498.582770:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734019498.582772:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:2.0:1734019498.582774:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000008:00100000:3.0:1734019498.582776:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019498.582780:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333167232:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019498.582786:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333168000/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019498.582787:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333167360:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019498.582807:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019498.582813:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019498/real 1734019498] req@ffff8800a73cc380 x1818247333167360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019498.582825:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333167360/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019498.582833:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019498.582836:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019498.582839:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333167360:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019498.582865:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333167488:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019498.582880:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019498.582886:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019498/real 1734019498] req@ffff8800a73cf100 x1818247333167488/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019498.582896:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333167488/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019498.582904:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019498.582908:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019498.582911:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333167488:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019498.582917:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333167616:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019498.582929:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019498.582935:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019498/real 1734019498] req@ffff8800a73cc700 x1818247333167616/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019498.582945:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333167616/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019498.582954:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019498.582956:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019498.582959:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333167616:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019498.582964:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333167744:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019498.582976:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019498.582982:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019498/real 1734019498] req@ffff8800a73cdf80 x1818247333167744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019498.582992:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333167744/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019498.582999:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019498.583001:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019498.583004:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333167744:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019498.583009:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333167872:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019498.583026:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019498.583032:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019498/real 1734019498] req@ffff8800a73cf800 x1818247333167872/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019498.583042:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333167872/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019498.583049:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019498.583051:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019498.583054:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333167872:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019498.583059:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333168000:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019498.583072:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019498.583077:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019498/real 1734019498] req@ffff8800a73cfb80 x1818247333168000/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019498.583087:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333168000/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019498.583094:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019498.583097:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019498.583100:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333168000:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019498.698490:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019499.700496:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019500.702543:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019501.704533:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019502.706542:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019503.590359:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019503.590368:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019503.590373:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019503.590383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3766 00000100:00080000:3.0:1734019503.590393:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019503.590401:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019503.590437:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333168128/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019503.590503:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019503.590508:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019503.590510:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019503.590515:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3766 00000100:00080000:3.0:1734019503.590518:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019503.590522:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019503.590532:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019503.590546:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333168256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019503.590557:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019503.590560:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019503.590562:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019503.590566:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3766 00000100:00080000:3.0:1734019503.590568:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019503.590571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019503.590577:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019503.590585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333168384/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019503.590594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019503.590598:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019503.590599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019503.590603:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3766 00000100:00080000:3.0:1734019503.590605:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019503.590608:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019503.590613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019503.590620:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333168512/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019503.590630:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734019503.590632:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333168128:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019503.590633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019503.590634:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019503.590639:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3766 00000100:00080000:3.0:1734019503.590641:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019503.590644:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019503.590649:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019503.590657:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333168640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019503.590665:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019503.590668:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019503.590670:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019503.590673:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3766 00000100:00080000:3.0:1734019503.590677:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019503.590680:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019503.590684:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019503.590691:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333168768/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019503.590699:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019503.590702:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019503.590704:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019503.590707:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3766 00000100:00080000:3.0:1734019503.590713:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019503.590716:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019503.590720:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019503.590726:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019503.590728:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333168896/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019503.590755:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019503/real 1734019503] req@ffff8800a73cdc00 x1818247333168128/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019503.590785:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333168128/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019503.590798:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019503.590802:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019503.590807:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333168128:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019503.590815:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333168256:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019503.590835:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019503.590841:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019503/real 1734019503] req@ffff8800a73cfb80 x1818247333168256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019503.590853:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333168256/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019503.590861:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019503.590864:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019503.590867:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333168256:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019503.590874:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333168384:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019503.590889:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019503.590895:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019503/real 1734019503] req@ffff8800a73cf800 x1818247333168384/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019503.590905:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333168384/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019503.590912:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019503.590916:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019503.590920:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333168384:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019503.590951:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333168512:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019503.590964:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019503.590971:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019503/real 1734019503] req@ffff8800a73cdf80 x1818247333168512/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019503.590981:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333168512/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019503.590989:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019503.590991:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019503.590994:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333168512:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019503.591000:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333168640:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019503.591012:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019503.591018:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019503/real 1734019503] req@ffff8800a73cc700 x1818247333168640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019503.591028:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333168640/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019503.591036:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019503.591038:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019503.591041:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333168640:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019503.591047:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333168768:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019503.591064:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019503.591070:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019503/real 1734019503] req@ffff8800a73cf100 x1818247333168768/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019503.591081:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333168768/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019503.591088:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019503.591090:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019503.591093:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333168768:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019503.591099:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333168896:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019503.591111:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019503.591118:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019503/real 1734019503] req@ffff8800a73cc380 x1818247333168896/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019503.591128:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333168896/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019503.591135:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019503.591138:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019503.591141:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333168896:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019503.708495:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019504.710472:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019505.712507:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019506.714495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019507.716455:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019508.598397:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019508.598406:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019508.598411:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019508.598422:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3771 00000100:00080000:3.0:1734019508.598432:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019508.598440:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019508.598476:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333169024/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019508.598509:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019508.598513:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019508.598515:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019508.598520:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3771 00000100:00080000:3.0:1734019508.598523:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019508.598527:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019508.598537:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019508.598548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333169152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019508.598558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019508.598561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019508.598563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019508.598566:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3771 00000100:00080000:3.0:1734019508.598569:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019508.598573:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019508.598577:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019508.598586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333169280/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019508.598594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019508.598597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019508.598599:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019508.598602:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3771 00000100:00080000:3.0:1734019508.598605:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019508.598609:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019508.598613:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019508.598614:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333169024:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019508.598621:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333169408/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019508.598630:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019508.598633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019508.598634:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019508.598638:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3771 00000100:00080000:3.0:1734019508.598641:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019508.598644:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019508.598648:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019508.598656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333169536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019508.598664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019508.598667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019508.598668:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019508.598672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3771 00000100:00080000:3.0:1734019508.598675:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019508.598678:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019508.598682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019508.598683:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019508.598690:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333169664/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019508.598699:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019508.598702:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019508.598704:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019508.598706:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019508/real 1734019508] req@ffff8800a73cf480 x1818247333169024/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019508.598707:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3771 00000100:00080000:3.0:1734019508.598710:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019508.598713:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019508.598717:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019508.598726:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333169792/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019508.598734:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333169024/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019508.598744:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019508.598748:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019508.598752:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333169024:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019508.598759:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333169152:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019508.598777:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019508.598783:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019508/real 1734019508] req@ffff8800a73cce00 x1818247333169152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019508.598794:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333169152/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019508.598802:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019508.598804:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019508.598807:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333169152:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019508.598813:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333169280:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019508.598826:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019508.598832:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019508/real 1734019508] req@ffff8800a73ced80 x1818247333169280/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019508.598842:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333169280/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019508.598849:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019508.598853:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019508.598856:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333169280:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019508.598882:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333169408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019508.598895:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019508.598901:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019508/real 1734019508] req@ffff8800a73cd880 x1818247333169408/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019508.598911:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333169408/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019508.598918:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019508.598920:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019508.598923:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333169408:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019508.598928:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333169536:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019508.598939:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019508.598945:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019508/real 1734019508] req@ffff8800a73cea00 x1818247333169536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019508.598954:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333169536/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019508.598962:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019508.598964:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019508.598967:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333169536:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019508.598972:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333169664:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019508.598987:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019508.598993:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019508/real 1734019508] req@ffff8800a73cca80 x1818247333169664/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019508.599003:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333169664/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019508.599010:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019508.599012:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019508.599015:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333169664:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019508.599019:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333169792:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019508.599031:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019508.599037:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019508/real 1734019508] req@ffff8800a73cd500 x1818247333169792/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019508.599046:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333169792/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019508.599053:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019508.599055:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019508.599058:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333169792:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019508.718519:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019509.720504:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019510.722505:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019511.724514:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019512.726495:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019513.606380:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019513.606389:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019513.606393:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019513.606403:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3776 00000100:00080000:3.0:1734019513.606411:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019513.606419:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019513.606454:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333169920/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019513.606486:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019513.606490:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019513.606493:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019513.606498:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3776 00000100:00080000:3.0:1734019513.606501:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019513.606504:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019513.606514:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019513.606526:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333170048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019513.606536:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019513.606539:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019513.606541:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019513.606544:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3776 00000100:00080000:3.0:1734019513.606547:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019513.606551:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019513.606555:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019513.606563:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333170176/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019513.606572:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019513.606575:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019513.606576:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019513.606580:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3776 00000100:00080000:3.0:1734019513.606582:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019513.606585:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019513.606590:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019513.606592:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333169920:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019513.606597:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333170304/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019513.606607:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019513.606610:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019513.606612:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019513.606615:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3776 00000100:00080000:3.0:1734019513.606617:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019513.606621:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019513.606625:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019513.606632:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333170432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019513.606640:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019513.606643:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019513.606645:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019513.606649:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3776 00000100:00080000:3.0:1734019513.606651:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019513.606654:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:2.0:1734019513.606659:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734019513.606659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019513.606666:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333170560/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019513.606674:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019513.606677:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019513.606679:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019513.606682:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019513/real 1734019513] req@ffff8800a73ce680 x1818247333169920/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019513.606683:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3776 00000100:00080000:3.0:1734019513.606686:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019513.606689:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019513.606694:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019513.606702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333170688/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019513.606710:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333169920/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019513.606720:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019513.606723:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019513.606728:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333169920:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019513.606735:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333170048:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019513.606752:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019513.606759:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019513/real 1734019513] req@ffff8800a73cd500 x1818247333170048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019513.606770:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333170048/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019513.606777:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019513.606780:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019513.606783:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333170048:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019513.606789:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333170176:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019513.606802:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019513.606808:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019513/real 1734019513] req@ffff8800a73cca80 x1818247333170176/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019513.606818:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333170176/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019513.606825:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019513.606829:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019513.606832:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333170176:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019513.606855:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333170304:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019513.606867:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019513.606873:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019513/real 1734019513] req@ffff8800a73cea00 x1818247333170304/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019513.606884:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333170304/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019513.606891:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019513.606893:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019513.606896:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333170304:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019513.606901:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333170432:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019513.606912:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019513.606918:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019513/real 1734019513] req@ffff8800a73cd880 x1818247333170432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019513.606927:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333170432/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019513.606934:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019513.606936:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019513.606940:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333170432:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019513.606945:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333170560:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019513.606960:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019513.606966:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019513/real 1734019513] req@ffff8800a73ced80 x1818247333170560/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019513.606976:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333170560/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019513.606983:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019513.606985:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019513.606988:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333170560:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019513.606993:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333170688:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019513.607004:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019513.607010:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019513/real 1734019513] req@ffff8800a73cce00 x1818247333170688/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019513.607019:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333170688/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019513.607026:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019513.607029:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019513.607031:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333170688:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019513.728509:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019514.730499:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019515.732476:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019516.734470:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019517.736467:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019518.614354:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019518.614362:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019518.614366:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019518.614377:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3781 00000100:00080000:3.0:1734019518.614386:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019518.614394:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019518.614429:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333170816/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019518.614461:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019518.614465:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019518.614467:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019518.614472:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3781 00000100:00080000:3.0:1734019518.614475:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019518.614479:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019518.614488:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019518.614500:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333170944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019518.614510:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019518.614514:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019518.614515:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019518.614519:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3781 00000100:00080000:3.0:1734019518.614521:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019518.614525:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019518.614529:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019518.614537:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333171072/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019518.614546:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019518.614549:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019518.614550:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019518.614553:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3781 00000100:00080000:3.0:1734019518.614556:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019518.614559:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019518.614564:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019518.614569:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333170816:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019518.614571:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333171200/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019518.614580:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019518.614583:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019518.614585:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019518.614589:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3781 00000100:00080000:3.0:1734019518.614591:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019518.614595:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019518.614599:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019518.614607:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333171328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019518.614615:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019518.614618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019518.614619:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019518.614623:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3781 00000100:00080000:3.0:1734019518.614626:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019518.614629:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019518.614633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019518.614635:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019518.614640:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333171456/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019518.614648:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019518.614651:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019518.614653:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019518.614656:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3781 00000100:00100000:2.0:1734019518.614657:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019518/real 1734019518] req@ffff8800a73cf480 x1818247333170816/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019518.614659:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019518.614662:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019518.614666:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019518.614675:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333171584/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019518.614685:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333170816/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019518.614695:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019518.614698:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019518.614703:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333170816:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019518.614710:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333170944:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019518.614726:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019518.614733:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019518/real 1734019518] req@ffff8800a73cc380 x1818247333170944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019518.614744:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333170944/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019518.614752:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019518.614755:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019518.614758:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333170944:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019518.614764:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333171072:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019518.614777:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019518.614783:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019518/real 1734019518] req@ffff8800a73cf100 x1818247333171072/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019518.614793:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333171072/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019518.614800:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019518.614803:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019518.614806:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333171072:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019518.614831:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333171200:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019518.614844:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019518.614850:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019518/real 1734019518] req@ffff8800a73cc700 x1818247333171200/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019518.614861:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333171200/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019518.614868:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019518.614870:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019518.614873:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333171200:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019518.614878:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333171328:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019518.614889:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019518.614895:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019518/real 1734019518] req@ffff8800a73cdf80 x1818247333171328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019518.614905:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333171328/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019518.614912:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019518.614914:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019518.614917:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333171328:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019518.614922:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333171456:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019518.614937:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019518.614943:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019518/real 1734019518] req@ffff8800a73cf800 x1818247333171456/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019518.614953:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333171456/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019518.614960:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019518.614962:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019518.614965:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333171456:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019518.614970:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333171584:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019518.614982:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019518.614987:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019518/real 1734019518] req@ffff8800a73cfb80 x1818247333171584/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019518.614997:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333171584/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019518.615004:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019518.615006:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019518.615009:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333171584:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019518.738491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019519.740468:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019520.742501:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019521.744488:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019522.746493:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019523.622360:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019523.622370:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019523.622374:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019523.622384:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3786 00000100:00080000:3.0:1734019523.622394:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019523.622402:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019523.622438:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333171712/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019523.622471:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019523.622475:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019523.622477:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019523.622482:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3786 00000100:00080000:3.0:1734019523.622485:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019523.622489:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019523.622499:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019523.622512:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333171840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019523.622522:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019523.622526:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019523.622528:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019523.622532:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3786 00000100:00080000:3.0:1734019523.622534:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019523.622537:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019523.622547:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019523.622555:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333171968/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019523.622563:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019523.622566:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019523.622568:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019523.622571:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3786 00000100:00080000:3.0:1734019523.622574:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019523.622577:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:2.0:1734019523.622580:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333171712:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734019523.622582:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019523.622589:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333172096/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019523.622597:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019523.622601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019523.622602:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019523.622606:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3786 00000100:00080000:3.0:1734019523.622608:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019523.622611:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019523.622615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019523.622623:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333172224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019523.622632:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019523.622635:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019523.622637:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019523.622641:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3786 00000100:00080000:3.0:1734019523.622644:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734019523.622647:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019523.622647:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019523.622651:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019523.622658:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333172352/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019523.622666:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1734019523.622670:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019523/real 1734019523] req@ffff8800a73cdc00 x1818247333171712/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019523.622670:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019523.622671:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019523.622674:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3786 00000100:00080000:3.0:1734019523.622677:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019523.622680:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019523.622685:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019523.622692:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333172480/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019523.622697:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333171712/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019523.622707:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019523.622710:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019523.622715:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333171712:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019523.622722:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333171840:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019523.622739:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019523.622746:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019523/real 1734019523] req@ffff8800a73cfb80 x1818247333171840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019523.622756:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333171840/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019523.622764:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019523.622767:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019523.622770:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333171840:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019523.622775:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333171968:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019523.622788:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019523.622794:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019523/real 1734019523] req@ffff8800a73cf800 x1818247333171968/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019523.622804:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333171968/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019523.622811:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019523.622815:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019523.622818:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333171968:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019523.622843:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333172096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019523.622855:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019523.622861:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019523/real 1734019523] req@ffff8800a73cdf80 x1818247333172096/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019523.622872:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333172096/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019523.622879:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019523.622881:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019523.622884:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333172096:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019523.622889:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333172224:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019523.622900:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019523.622906:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019523/real 1734019523] req@ffff8800a73cc700 x1818247333172224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019523.622916:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333172224/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019523.622923:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019523.622925:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019523.622928:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333172224:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019523.622933:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333172352:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019523.622948:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019523.622954:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019523/real 1734019523] req@ffff8800a73cf100 x1818247333172352/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019523.622965:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333172352/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019523.622972:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019523.622974:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019523.622977:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333172352:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019523.622981:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333172480:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019523.622993:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019523.622998:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019523/real 1734019523] req@ffff8800a73cc380 x1818247333172480/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019523.623008:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333172480/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019523.623015:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019523.623017:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019523.623020:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333172480:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019523.748448:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019524.750497:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019525.752464:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019526.754467:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019527.756420:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019528.630348:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019528.630357:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019528.630362:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019528.630372:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3791 00000100:00080000:3.0:1734019528.630382:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019528.630390:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019528.630426:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333172608/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019528.630459:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019528.630463:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019528.630465:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019528.630470:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3791 00000100:00080000:3.0:1734019528.630473:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019528.630477:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019528.630487:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019528.630498:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333172736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019528.630507:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019528.630510:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019528.630512:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019528.630516:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3791 00000100:00080000:3.0:1734019528.630519:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019528.630522:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019528.630527:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019528.630535:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333172864/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019528.630544:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019528.630548:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019528.630549:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019528.630552:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3791 00000100:00080000:3.0:1734019528.630555:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019528.630558:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:2.0:1734019528.630563:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333172608:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734019528.630563:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019528.630570:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333172992/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019528.630579:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019528.630582:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019528.630584:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019528.630587:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3791 00000100:00080000:3.0:1734019528.630589:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019528.630593:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019528.630597:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019528.630604:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333173120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019528.630612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019528.630615:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019528.630616:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019528.630620:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3791 00000100:00080000:3.0:1734019528.630623:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019528.630627:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:2.0:1734019528.630630:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734019528.630631:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019528.630639:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333173248/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019528.630647:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019528.630650:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1734019528.630652:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019528/real 1734019528] req@ffff8800a73cf480 x1818247333172608/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019528.630652:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019528.630655:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3791 00000100:00080000:3.0:1734019528.630658:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019528.630661:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019528.630665:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019528.630674:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333173376/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019528.630679:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333172608/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019528.630689:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019528.630692:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019528.630697:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333172608:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019528.630705:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333172736:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019528.630722:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019528.630729:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019528/real 1734019528] req@ffff8800a73cce00 x1818247333172736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019528.630740:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333172736/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019528.630747:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019528.630750:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019528.630753:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333172736:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019528.630759:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333172864:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019528.630772:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019528.630778:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019528/real 1734019528] req@ffff8800a73ced80 x1818247333172864/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019528.630787:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333172864/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019528.630795:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019528.630798:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019528.630801:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333172864:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019528.630827:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333172992:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019528.630839:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019528.630846:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019528/real 1734019528] req@ffff8800a73cd880 x1818247333172992/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019528.630856:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333172992/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019528.630863:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019528.630866:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019528.630869:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333172992:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019528.630873:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333173120:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019528.630885:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019528.630890:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019528/real 1734019528] req@ffff8800a73cea00 x1818247333173120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019528.630900:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333173120/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019528.630907:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019528.630910:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019528.630913:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333173120:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019528.630918:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333173248:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019528.630933:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019528.630939:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019528/real 1734019528] req@ffff8800a73cca80 x1818247333173248/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019528.630949:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333173248/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019528.630957:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019528.630959:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019528.630962:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333173248:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019528.630967:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333173376:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019528.630978:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019528.630984:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019528/real 1734019528] req@ffff8800a73cd500 x1818247333173376/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019528.630993:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333173376/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019528.631000:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019528.631003:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019528.631005:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333173376:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019528.758487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019529.760436:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019530.762486:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019531.764464:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019532.766485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019533.638365:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019533.638374:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019533.638378:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019533.638388:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3796 00000100:00080000:3.0:1734019533.638398:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019533.638406:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019533.638441:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333173504/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019533.638475:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019533.638479:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019533.638481:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019533.638486:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3796 00000100:00080000:3.0:1734019533.638489:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019533.638493:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019533.638503:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019533.638516:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333173632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019533.638526:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019533.638529:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019533.638531:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019533.638534:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3796 00000100:00080000:3.0:1734019533.638537:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019533.638540:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019533.638545:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019533.638553:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333173760/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019533.638562:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019533.638565:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019533.638566:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019533.638570:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3796 00000100:00080000:3.0:1734019533.638572:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019533.638576:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019533.638580:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019533.638586:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333173504:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019533.638588:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333173888/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019533.638596:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019533.638600:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019533.638601:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019533.638604:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3796 00000100:00080000:3.0:1734019533.638607:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019533.638610:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019533.638614:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019533.638622:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333174016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019533.638630:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019533.638633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019533.638635:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019533.638638:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3796 00000100:00080000:3.0:1734019533.638642:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019533.638646:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019533.638650:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019533.638652:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019533.638658:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333174144/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019533.638666:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019533.638670:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019533.638671:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019533.638675:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019533/real 1734019533] req@ffff8800a73ce680 x1818247333173504/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019533.638675:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3796 00000100:00080000:3.0:1734019533.638677:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019533.638681:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019533.638685:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019533.638692:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333174272/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019533.638703:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333173504/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019533.638713:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019533.638716:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019533.638720:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333173504:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019533.638727:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333173632:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019533.638745:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019533.638751:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019533/real 1734019533] req@ffff8800a73cd500 x1818247333173632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019533.638762:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333173632/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019533.638770:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019533.638772:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019533.638775:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333173632:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019533.638781:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333173760:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019533.638794:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019533.638799:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019533/real 1734019533] req@ffff8800a73cca80 x1818247333173760/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019533.638809:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333173760/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019533.638817:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019533.638820:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019533.638823:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333173760:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019533.638848:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333173888:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019533.638860:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019533.638866:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019533/real 1734019533] req@ffff8800a73cea00 x1818247333173888/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019533.638877:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333173888/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019533.638884:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019533.638886:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019533.638889:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333173888:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019533.638894:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333174016:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019533.638905:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019533.638910:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019533/real 1734019533] req@ffff8800a73cd880 x1818247333174016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019533.638920:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333174016/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019533.638927:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019533.638929:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019533.638933:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333174016:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019533.638985:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333174144:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019533.639006:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019533.639014:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019533/real 1734019533] req@ffff8800a73ced80 x1818247333174144/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019533.639025:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333174144/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019533.639032:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019533.639034:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019533.639037:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333174144:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019533.639043:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333174272:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019533.639054:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019533.639060:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019533/real 1734019533] req@ffff8800a73cce00 x1818247333174272/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019533.639070:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333174272/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019533.639077:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019533.639080:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019533.639083:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333174272:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019533.768489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019534.770481:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019535.772500:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019536.774498:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019537.776496:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019538.646357:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019538.646367:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019538.646371:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019538.646382:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3801 00000100:00080000:3.0:1734019538.646392:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019538.646400:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019538.646437:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333174400/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019538.646469:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019538.646473:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019538.646476:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019538.646481:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3801 00000100:00080000:3.0:1734019538.646484:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019538.646488:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019538.646499:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019538.646509:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333174528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019538.646519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019538.646523:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019538.646524:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019538.646528:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3801 00000100:00080000:3.0:1734019538.646531:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019538.646534:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019538.646539:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019538.646547:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333174656/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019538.646555:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019538.646558:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019538.646560:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019538.646563:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3801 00000100:00080000:3.0:1734019538.646566:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019538.646569:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019538.646573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019538.646576:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333174400:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019538.646581:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333174784/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019538.646590:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019538.646593:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019538.646595:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019538.646598:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3801 00000100:00080000:3.0:1734019538.646601:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019538.646604:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019538.646608:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019538.646616:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333174912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019538.646624:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019538.646627:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019538.646628:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019538.646632:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3801 00000100:00080000:3.0:1734019538.646635:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019538.646638:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019538.646642:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019538.646644:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019538.646650:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333175040/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019538.646658:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019538.646661:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019538.646663:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019538.646666:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019538/real 1734019538] req@ffff8800a73cf480 x1818247333174400/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019538.646666:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3801 00000100:00080000:3.0:1734019538.646670:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019538.646673:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019538.646677:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019538.646686:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333175168/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019538.646693:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333174400/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019538.646703:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019538.646706:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019538.646711:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333174400:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019538.646718:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333174528:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019538.646735:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019538.646742:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019538/real 1734019538] req@ffff8800a73cc380 x1818247333174528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019538.646753:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333174528/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019538.646760:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019538.646763:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019538.646766:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333174528:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019538.646772:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333174656:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019538.646785:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019538.646791:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019538/real 1734019538] req@ffff8800a73cf100 x1818247333174656/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019538.646801:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333174656/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019538.646808:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019538.646811:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019538.646815:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333174656:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019538.646839:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333174784:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019538.646852:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019538.646904:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019538/real 1734019538] req@ffff8800a73cc700 x1818247333174784/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019538.646916:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333174784/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019538.646923:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019538.646926:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019538.646929:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333174784:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019538.646934:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333174912:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019538.646950:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019538.646956:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019538/real 1734019538] req@ffff8800a73cdf80 x1818247333174912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019538.646966:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333174912/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019538.646973:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019538.646976:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019538.646979:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333174912:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019538.646983:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333175040:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019538.646998:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019538.647004:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019538/real 1734019538] req@ffff8800a73cf800 x1818247333175040/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019538.647014:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333175040/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019538.647022:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019538.647024:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019538.647027:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333175040:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019538.647032:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333175168:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019538.647043:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019538.647049:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019538/real 1734019538] req@ffff8800a73cfb80 x1818247333175168/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019538.647058:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333175168/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019538.647065:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019538.647068:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019538.647071:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333175168:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019538.778484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019539.780477:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019540.782473:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019541.784483:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019542.786506:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019543.654370:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019543.654379:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019543.654383:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019543.654394:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3806 00000100:00080000:3.0:1734019543.654404:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019543.654411:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019543.654448:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333175296/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019543.654481:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019543.654486:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019543.654488:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019543.654492:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3806 00000100:00080000:3.0:1734019543.654496:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019543.654499:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019543.654509:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019543.654522:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333175424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019543.654532:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019543.654535:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019543.654537:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019543.654541:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3806 00000100:00080000:3.0:1734019543.654543:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019543.654546:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019543.654551:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019543.654559:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333175552/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019543.654568:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019543.654571:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019543.654572:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019543.654576:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3806 00000100:00080000:3.0:1734019543.654578:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019543.654581:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019543.654586:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019543.654588:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333175296:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019543.654593:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333175680/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019543.654602:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019543.654605:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019543.654606:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019543.654610:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3806 00000100:00080000:3.0:1734019543.654613:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019543.654616:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019543.654620:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019543.654628:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333175808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019543.654636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019543.654639:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019543.654640:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019543.654644:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3806 00000100:00080000:3.0:1734019543.654647:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019543.654650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019543.654654:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019543.654655:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019543.654661:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333175936/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019543.654669:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019543.654672:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019543.654674:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019543.654676:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019543/real 1734019543] req@ffff8800a73cdc00 x1818247333175296/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019543.654677:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3806 00000100:00080000:3.0:1734019543.654680:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019543.654683:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019543.654688:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019543.654696:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333176064/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019543.654703:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333175296/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019543.654714:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019543.654717:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019543.654722:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333175296:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019543.654729:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333175424:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019543.654746:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019543.654752:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019543/real 1734019543] req@ffff8800a73cfb80 x1818247333175424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019543.654764:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333175424/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019543.654819:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019543.654822:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019543.654826:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333175424:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019543.654838:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333175552:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019543.654855:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019543.654863:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019543/real 1734019543] req@ffff8800a73cf800 x1818247333175552/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019543.654873:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333175552/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019543.654881:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019543.654884:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019543.654887:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333175552:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019543.654911:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333175680:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019543.654924:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019543.654930:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019543/real 1734019543] req@ffff8800a73cdf80 x1818247333175680/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019543.654940:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333175680/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019543.654948:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019543.654950:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019543.654953:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333175680:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019543.654957:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333175808:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019543.654969:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019543.654975:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019543/real 1734019543] req@ffff8800a73cc700 x1818247333175808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019543.654984:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333175808/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019543.654991:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019543.654993:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019543.654997:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333175808:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019543.655002:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333175936:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019543.655018:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019543.655024:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019543/real 1734019543] req@ffff8800a73cf100 x1818247333175936/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019543.655034:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333175936/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019543.655041:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019543.655043:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019543.655046:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333175936:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019543.655051:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333176064:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019543.655063:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019543.655069:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019543/real 1734019543] req@ffff8800a73cc380 x1818247333176064/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019543.655079:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333176064/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019543.655086:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019543.655088:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019543.655091:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333176064:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019543.788463:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019544.790475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019545.792420:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019546.794446:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019547.796434:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019548.662361:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019548.662372:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019548.662378:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019548.662391:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3811 00000100:00080000:3.0:1734019548.662403:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019548.662413:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019548.662462:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333176192/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019548.662499:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019548.662504:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019548.662506:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019548.662511:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3811 00000100:00080000:3.0:1734019548.662515:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019548.662519:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019548.662531:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019548.662543:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333176320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019548.662553:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019548.662556:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019548.662559:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019548.662563:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3811 00000100:00080000:3.0:1734019548.662566:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019548.662570:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019548.662575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019548.662585:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333176448/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019548.662593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019548.662597:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019548.662598:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019548.662602:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3811 00000100:00080000:3.0:1734019548.662604:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019548.662608:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019548.662612:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019548.662622:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333176576/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019548.662625:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333176192:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019548.662632:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019548.662635:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019548.662637:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019548.662640:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3811 00000100:00080000:3.0:1734019548.662643:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019548.662646:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019548.662651:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019548.662660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333176704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019548.662669:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019548.662672:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019548.662674:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019548.662678:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3811 00000100:00080000:3.0:1734019548.662736:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019548.662741:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019548.662746:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019548.662747:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019548.662756:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333176832/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019548.662766:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019548.662770:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1734019548.662772:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019548/real 1734019548] req@ffff8800a73cf480 x1818247333176192/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019548.662772:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019548.662776:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3811 00000100:00080000:3.0:1734019548.662779:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019548.662783:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019548.662787:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019548.662798:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333176960/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019548.662800:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333176192/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019548.662813:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019548.662816:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019548.662822:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333176192:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019548.662831:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333176320:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019548.662851:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019548.662858:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019548/real 1734019548] req@ffff8800a73cce00 x1818247333176320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019548.662869:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333176320/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019548.662877:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019548.662880:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019548.662883:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333176320:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019548.662889:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333176448:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019548.662903:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019548.662909:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019548/real 1734019548] req@ffff8800a73ced80 x1818247333176448/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019548.662919:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333176448/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019548.662927:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019548.662930:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019548.662933:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333176448:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019548.662961:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333176576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019548.662974:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019548.662980:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019548/real 1734019548] req@ffff8800a73cd880 x1818247333176576/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019548.662991:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333176576/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019548.662999:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019548.663001:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019548.663004:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333176576:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019548.663009:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333176704:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019548.663021:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019548.663027:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019548/real 1734019548] req@ffff8800a73cea00 x1818247333176704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019548.663037:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333176704/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019548.663044:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019548.663047:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019548.663050:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333176704:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019548.663056:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333176832:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019548.663071:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019548.663077:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019548/real 1734019548] req@ffff8800a73cca80 x1818247333176832/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019548.663088:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333176832/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019548.663095:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019548.663097:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019548.663100:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333176832:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019548.663105:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333176960:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019548.663117:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019548.663123:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019548/real 1734019548] req@ffff8800a73cd500 x1818247333176960/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019548.663133:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333176960/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019548.663140:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019548.663142:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019548.663145:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333176960:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019548.798508:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019549.800485:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019550.802491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019551.804491:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019552.806484:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019553.670341:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019553.670351:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019553.670355:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019553.670366:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3816 00000100:00080000:3.0:1734019553.670375:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019553.670383:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019553.670418:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333177088/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019553.670452:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019553.670456:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019553.670458:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019553.670462:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3816 00000100:00080000:3.0:1734019553.670465:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019553.670469:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019553.670479:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019553.670491:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333177216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019553.670500:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019553.670504:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019553.670506:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019553.670509:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3816 00000100:00080000:3.0:1734019553.670512:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019553.670515:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019553.670520:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019553.670529:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333177344/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019553.670538:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019553.670541:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019553.670543:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019553.670546:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3816 00000100:00080000:3.0:1734019553.670549:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019553.670552:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019553.670556:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019553.670559:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333177088:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019553.670564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333177472/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019553.670573:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019553.670576:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019553.670577:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019553.670581:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3816 00000100:00080000:3.0:1734019553.670584:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019553.670587:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019553.670591:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019553.670598:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333177600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019553.670643:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019553.670646:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019553.670648:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019553.670652:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3816 00000100:00080000:3.0:1734019553.670655:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019553.670659:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019553.670664:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019553.670673:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333177728/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019553.670682:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019553.670685:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019553.670687:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019553.670689:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333177216:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734019553.670690:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3816 00000100:00080000:3.0:1734019553.670692:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019553.670696:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019553.670700:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019553.670708:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333177856/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019553.670711:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333177344:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019553.670734:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333177472:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019553.670749:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333177600:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019553.670762:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333177728:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019553.670777:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333177856:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019553.808471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019554.810462:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019555.812439:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1734019556.676500:0:1955:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:0.0:1734019556.676548:0:1955:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1734019556.676603:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019556.676644:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019556.676652:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019556.676659:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019556.676666:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019556.676672:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:0.0:1734019556.676678:0:1955:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:3.0:1734019556.676774:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019553/real 1734019556] req@ffff8800a73cea00 x1818247333177472/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019556.676813:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333177472/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019556.676831:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734019556.676837:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734019556.676845:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333177472:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734019556.676858:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019553/real 1734019556] req@ffff8800a73cd880 x1818247333177600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019556.676878:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333177600/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019556.676891:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734019556.676895:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734019556.676901:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333177600:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734019556.676910:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019553/real 1734019556] req@ffff8800a73ced80 x1818247333177728/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019556.676929:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333177728/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019556.676941:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734019556.676946:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734019556.676952:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333177728:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734019556.676962:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019553/real 1734019556] req@ffff8800a73cce00 x1818247333177856/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019556.676981:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333177856/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019556.676994:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734019556.677009:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734019556.677015:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333177856:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734019556.677031:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019553/real 1734019556] req@ffff8800a73ce680 x1818247333177088/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019556.677050:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333177088/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019556.677062:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734019556.677066:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734019556.677071:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333177088:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019556.677081:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019553/real 1734019556] req@ffff8800a73cd500 x1818247333177216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019556.677097:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333177216/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019556.677110:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734019556.677114:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734019556.677119:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333177216:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734019556.677129:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019553/real 1734019556] req@ffff8800a73cca80 x1818247333177344/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019556.677146:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333177344/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019556.677158:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734019556.677162:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734019556.677167:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333177344:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019556.814516:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019557.816471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019558.678338:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019558.678348:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019558.678353:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019558.678363:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3821 00000100:00080000:3.0:1734019558.678373:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019558.678380:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019558.678416:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333177984/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019558.678450:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019558.678454:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019558.678456:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019558.678461:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3821 00000100:00080000:3.0:1734019558.678464:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019558.678468:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019558.678478:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019558.678488:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333178112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019558.678499:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019558.678502:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019558.678503:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019558.678507:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3821 00000100:00080000:3.0:1734019558.678510:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019558.678513:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019558.678559:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019558.678569:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333178240/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019558.678579:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019558.678583:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019558.678584:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734019558.678587:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333177984:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019558.678588:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3821 00000100:00080000:3.0:1734019558.678591:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019558.678594:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019558.678599:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019558.678606:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333178368/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019558.678615:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019558.678618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019558.678620:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019558.678623:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3821 00000100:00080000:3.0:1734019558.678626:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019558.678629:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019558.678633:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019558.678640:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333178496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019558.678649:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019558.678652:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019558.678653:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1734019558.678654:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019558.678658:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3821 00000100:00080000:3.0:1734019558.678661:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019558.678664:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019558.678668:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019558.678675:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019558/real 1734019558] req@ffff8800a73cca80 x1818247333177984/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019558.678676:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333178624/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019558.678683:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019558.678687:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019558.678688:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019558.678691:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3821 00000100:00100000:0.0:1734019558.678694:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333177984/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019558.678694:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019558.678698:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019558.678702:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1734019558.678705:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1734019558.678708:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333178752/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019558.678709:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019558.678714:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333177984:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019558.678721:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333178112:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019558.678739:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019558.678746:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019558/real 1734019558] req@ffff8800a73cd500 x1818247333178112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019558.678757:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333178112/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019558.678765:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019558.678767:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019558.678771:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333178112:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019558.678793:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333178240:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019558.678807:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019558.678813:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019558/real 1734019558] req@ffff8800a73ce680 x1818247333178240/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019558.678823:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333178240/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019558.678830:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019558.678832:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019558.678836:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333178240:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019558.678841:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333178368:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019558.678853:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019558.678859:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019558/real 1734019558] req@ffff8800a73cce00 x1818247333178368/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019558.678875:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333178368/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019558.678883:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019558.678885:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019558.678888:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333178368:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019558.678894:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333178496:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019558.678906:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019558.678911:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019558/real 1734019558] req@ffff8800a73ced80 x1818247333178496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019558.678921:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333178496/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019558.678928:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019558.678931:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019558.678933:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333178496:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019558.678939:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333178624:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019558.678951:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019558.678957:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019558/real 1734019558] req@ffff8800a73cd880 x1818247333178624/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019558.678967:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333178624/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019558.678974:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019558.678977:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019558.678980:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333178624:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019558.678986:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333178752:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019558.678999:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019558.679005:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019558/real 1734019558] req@ffff8800a73cea00 x1818247333178752/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019558.679015:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333178752/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019558.679022:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019558.679024:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019558.679028:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333178752:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019558.818510:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019559.820555:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019560.822528:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019561.824530:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019562.826515:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019563.686357:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019563.686366:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019563.686370:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019563.686381:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3826 00000100:00080000:3.0:1734019563.686392:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019563.686399:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019563.686475:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333178880/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019563.686508:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019563.686512:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019563.686514:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019563.686519:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3826 00000100:00080000:3.0:1734019563.686522:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019563.686526:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019563.686536:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019563.686548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333179008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019563.686558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019563.686561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019563.686563:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019563.686567:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3826 00000100:00080000:3.0:1734019563.686569:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019563.686572:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019563.686578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019563.686586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333179136/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019563.686595:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019563.686600:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019563.686602:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019563.686606:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3826 00000100:00080000:3.0:1734019563.686608:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019563.686612:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019563.686616:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019563.686625:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333179264/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019563.686634:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333178880:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019563.686634:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019563.686637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019563.686639:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019563.686642:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3826 00000100:00080000:3.0:1734019563.686644:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019563.686648:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019563.686652:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019563.686660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333179392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019563.686668:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019563.686671:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019563.686673:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019563.686677:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3826 00000100:00080000:3.0:1734019563.686679:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019563.686683:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019563.686688:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019563.686695:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333179520/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019563.686703:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019563.686707:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019563.686708:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019563.686711:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3826 00000100:00080000:3.0:1734019563.686715:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019563.686718:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019563.686722:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734019563.686725:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019563.686731:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333179648/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019563.686754:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019563/real 1734019563] req@ffff8800a73cf480 x1818247333178880/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019563.686777:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333178880/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019563.686790:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019563.686794:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019563.686799:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333178880:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019563.686808:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333179008:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019563.686832:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019563.686839:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019563/real 1734019563] req@ffff8800a73cc380 x1818247333179008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019563.686851:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333179008/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019563.686859:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019563.686862:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019563.686866:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333179008:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019563.686872:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333179136:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019563.686887:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019563.686894:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019563/real 1734019563] req@ffff8800a73cf100 x1818247333179136/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019563.686904:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333179136/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019563.686911:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019563.686914:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019563.686917:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333179136:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019563.686948:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333179264:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019563.686961:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019563.686968:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019563/real 1734019563] req@ffff8800a73cc700 x1818247333179264/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019563.686986:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333179264/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019563.686994:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019563.686996:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019563.686999:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333179264:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019563.687005:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333179392:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019563.687018:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019563.687024:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019563/real 1734019563] req@ffff8800a73cdf80 x1818247333179392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019563.687033:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333179392/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019563.687041:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019563.687043:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019563.687046:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333179392:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019563.687053:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333179520:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019563.687066:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019563.687072:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019563/real 1734019563] req@ffff8800a73cf800 x1818247333179520/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019563.687082:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333179520/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019563.687089:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019563.687093:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019563.687096:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333179520:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019563.687102:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333179648:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019563.687116:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019563.687122:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019563/real 1734019563] req@ffff8800a73cfb80 x1818247333179648/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019563.687132:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333179648/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019563.687139:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019563.687141:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019563.687144:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333179648:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019563.828510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019564.830494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019565.832499:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019566.834492:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019567.836458:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019568.694323:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019568.694340:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019568.694347:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019568.694402:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3831 00000100:00080000:3.0:1734019568.694414:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019568.694423:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019568.694473:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333179776/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019568.694519:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019568.694523:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019568.694525:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019568.694531:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3831 00000100:00080000:3.0:1734019568.694534:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019568.694538:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019568.694551:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019568.694565:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333179904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019568.694575:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019568.694579:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019568.694581:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019568.694585:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3831 00000100:00080000:3.0:1734019568.694588:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019568.694591:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019568.694596:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019568.694605:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333180032/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019568.694615:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019568.694618:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019568.694620:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019568.694624:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3831 00000100:00080000:3.0:1734019568.694626:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019568.694631:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019568.694632:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333179776:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734019568.694636:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019568.694644:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333180160/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019568.694653:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019568.694656:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019568.694657:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019568.694661:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3831 00000100:00080000:3.0:1734019568.694663:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019568.694667:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019568.694672:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019568.694680:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333180288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019568.694688:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019568.694691:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019568.694693:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019568.694697:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3831 00000100:00080000:3.0:1734019568.694709:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019568.694713:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019568.694717:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734019568.694719:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019568.694726:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333180416/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019568.694735:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019568.694739:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019568.694741:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019568.694744:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3831 00000100:00100000:0.0:1734019568.694747:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019568/real 1734019568] req@ffff8800a73cdc00 x1818247333179776/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019568.694748:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019568.694751:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019568.694756:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019568.694764:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333180544/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019568.694769:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333179776/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019568.694780:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019568.694784:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019568.694788:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333179776:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019568.694796:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333179904:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019568.694819:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019568.694826:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019568/real 1734019568] req@ffff8800a73cfb80 x1818247333179904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019568.694838:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333179904/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019568.694846:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019568.694849:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019568.694852:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333179904:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019568.694879:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333180032:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019568.694894:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019568.694900:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019568/real 1734019568] req@ffff8800a73cf800 x1818247333180032/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019568.694910:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333180032/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019568.694918:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019568.694920:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019568.694924:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333180032:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019568.694929:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333180160:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019568.694941:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019568.694947:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019568/real 1734019568] req@ffff8800a73cdf80 x1818247333180160/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019568.694965:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333180160/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019568.694973:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019568.694975:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019568.694978:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333180160:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019568.694983:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333180288:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019568.694995:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019568.695001:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019568/real 1734019568] req@ffff8800a73cc700 x1818247333180288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019568.695011:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333180288/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019568.695018:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019568.695020:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019568.695023:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333180288:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019568.695029:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333180416:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019568.695042:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019568.695048:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019568/real 1734019568] req@ffff8800a73cf100 x1818247333180416/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019568.695058:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333180416/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019568.695065:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019568.695068:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019568.695071:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333180416:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019568.695077:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333180544:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019568.695090:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019568.695096:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019568/real 1734019568] req@ffff8800a73cc380 x1818247333180544/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019568.695105:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333180544/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019568.695113:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019568.695115:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019568.695118:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333180544:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019568.838460:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019569.840476:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019570.842532:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019571.844524:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019572.846513:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019573.702389:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019573.702401:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019573.702407:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019573.702421:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3836 00000100:00080000:3.0:1734019573.702433:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019573.702442:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019573.702490:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333180672/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019573.702526:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019573.702531:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019573.702533:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019573.702538:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3836 00000100:00080000:3.0:1734019573.702543:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019573.702547:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019573.702559:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019573.702572:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333180800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019573.702582:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019573.702586:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019573.702588:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019573.702592:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3836 00000100:00080000:3.0:1734019573.702594:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019573.702598:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019573.702602:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019573.702611:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333180928/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019573.702621:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019573.702625:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019573.702626:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019573.702630:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3836 00000100:00080000:3.0:1734019573.702632:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019573.702636:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019573.702639:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333180672:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734019573.702641:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019573.702649:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333181056/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019573.702664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019573.702668:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019573.702670:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019573.702673:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3836 00000100:00080000:3.0:1734019573.702676:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019573.702679:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019573.702684:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019573.702692:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333181184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019573.702700:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019573.702703:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019573.702705:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019573.702709:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3836 00000100:00080000:3.0:1734019573.702713:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019573.702716:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019573.702721:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734019573.702726:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019573.702729:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333181312/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019573.702737:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019573.702741:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019573.702742:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019573.702746:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3836 00000100:00080000:3.0:1734019573.702750:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734019573.702753:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019573/real 1734019573] req@ffff8800a73cf480 x1818247333180672/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019573.702754:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019573.702758:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019573.702769:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333181440/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019573.702776:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333180672/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019573.702788:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019573.702792:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019573.702797:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333180672:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019573.702804:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333180800:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019573.702827:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019573.702834:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019573/real 1734019573] req@ffff8800a73cea00 x1818247333180800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019573.702846:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333180800/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019573.702854:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019573.702857:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019573.702860:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333180800:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019573.702888:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333180928:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019573.702903:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019573.702909:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019573/real 1734019573] req@ffff8800a73cd880 x1818247333180928/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019573.702919:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333180928/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019573.702927:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019573.702929:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019573.702933:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333180928:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019573.702938:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333181056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019573.702950:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019573.702956:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019573/real 1734019573] req@ffff8800a73ced80 x1818247333181056/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019573.702973:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333181056/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019573.702981:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019573.702983:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019573.702986:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333181056:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019573.702991:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333181184:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019573.703004:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019573.703010:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019573/real 1734019573] req@ffff8800a73cce00 x1818247333181184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019573.703020:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333181184/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019573.703027:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019573.703029:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019573.703032:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333181184:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019573.703038:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333181312:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019573.703051:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019573.703057:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019573/real 1734019573] req@ffff8800a73ce680 x1818247333181312/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019573.703067:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333181312/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019573.703075:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019573.703078:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019573.703081:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333181312:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019573.703087:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333181440:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019573.703100:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019573.703106:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019573/real 1734019573] req@ffff8800a73cd500 x1818247333181440/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019573.703115:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333181440/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019573.703123:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019573.703125:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019573.703129:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333181440:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019573.848510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019574.850489:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019575.852469:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019576.854518:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019577.856494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019578.710362:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019578.710371:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019578.710376:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019578.710387:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3841 00000100:00080000:3.0:1734019578.710396:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019578.710405:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019578.710441:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333181568/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019578.710474:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019578.710479:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019578.710481:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019578.710486:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3841 00000100:00080000:3.0:1734019578.710489:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019578.710493:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019578.710503:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019578.710516:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333181696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019578.710526:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019578.710530:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019578.710532:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019578.710535:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3841 00000100:00080000:3.0:1734019578.710538:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019578.710541:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019578.710546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019578.710554:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333181824/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019578.710562:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019578.710565:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019578.710567:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019578.710570:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3841 00000100:00080000:3.0:1734019578.710573:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019578.710576:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019578.710580:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019578.710588:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333181952/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019578.710589:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333181568:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019578.710596:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019578.710600:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019578.710601:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019578.710605:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3841 00000100:00080000:3.0:1734019578.710608:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019578.710611:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019578.710615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019578.710623:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333182080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019578.710631:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019578.710634:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019578.710635:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019578.710639:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3841 00000100:00080000:3.0:1734019578.710643:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019578.710646:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019578.710650:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019578.710658:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333182208/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734019578.710659:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019578.710665:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019578.710669:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019578.710670:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019578.710674:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3841 00000100:00080000:3.0:1734019578.710676:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734019578.710680:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019578/real 1734019578] req@ffff8800a73cca80 x1818247333181568/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019578.710680:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019578.710684:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019578.710691:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333182336/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019578.710699:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333181568/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019578.710710:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019578.710713:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019578.710718:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333181568:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019578.710725:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333181696:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019578.710745:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019578.710752:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019578/real 1734019578] req@ffff8800a73cd500 x1818247333181696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019578.710764:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333181696/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019578.710771:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019578.710774:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019578.710777:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333181696:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019578.710783:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333181824:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019578.710797:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019578.710802:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019578/real 1734019578] req@ffff8800a73ce680 x1818247333181824/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019578.710812:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333181824/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019578.710820:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019578.710822:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019578.710825:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333181824:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019578.710849:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333181952:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019578.710862:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019578.710868:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019578/real 1734019578] req@ffff8800a73cce00 x1818247333181952/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019578.710885:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333181952/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019578.710893:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019578.710895:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019578.710898:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333181952:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019578.710903:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333182080:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019578.710915:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019578.710920:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019578/real 1734019578] req@ffff8800a73ced80 x1818247333182080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019578.710930:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333182080/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019578.710938:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019578.710940:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019578.710943:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333182080:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019578.710948:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333182208:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019578.710960:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019578.710966:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019578/real 1734019578] req@ffff8800a73cd880 x1818247333182208/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019578.710976:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333182208/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019578.710982:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019578.710985:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019578.710988:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333182208:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019578.710994:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333182336:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019578.711006:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019578.711012:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019578/real 1734019578] req@ffff8800a73cea00 x1818247333182336/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019578.711022:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333182336/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019578.711029:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019578.711031:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019578.711034:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333182336:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019578.858492:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019579.860481:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019580.862494:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019581.864463:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019582.866507:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019583.718359:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019583.718368:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019583.718372:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019583.718383:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3846 00000100:00080000:3.0:1734019583.718393:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019583.718400:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019583.718437:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333182464/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019583.718472:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019583.718476:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019583.718478:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019583.718483:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3846 00000100:00080000:3.0:1734019583.718486:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019583.718490:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019583.718500:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019583.718511:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333182592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019583.718521:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019583.718524:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019583.718526:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019583.718530:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3846 00000100:00080000:3.0:1734019583.718533:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019583.718536:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019583.718541:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019583.718549:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333182720/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019583.718557:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019583.718560:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019583.718562:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019583.718565:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3846 00000100:00080000:3.0:1734019583.718568:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019583.718571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019583.718575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019583.718583:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333182464:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019583.718583:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333182848/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019583.718591:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019583.718595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019583.718596:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019583.718600:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3846 00000100:00080000:3.0:1734019583.718602:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019583.718605:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019583.718610:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019583.718618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333182976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019583.718626:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019583.718630:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019583.718631:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019583.718635:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3846 00000100:00080000:3.0:1734019583.718640:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019583.718643:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019583.718647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734019583.718651:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019583.718655:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333183104/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019583.718663:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019583.718666:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019583.718667:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019583.718670:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3846 00000100:00080000:3.0:1734019583.718673:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734019583.718674:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019583/real 1734019583] req@ffff8800a73cf480 x1818247333182464/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019583.718676:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019583.718681:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019583.718689:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333183232/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019583.718694:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333182464/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019583.718705:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019583.718708:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019583.718713:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333182464:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019583.718720:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333182592:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019583.718740:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019583.718748:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019583/real 1734019583] req@ffff8800a73cc380 x1818247333182592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019583.718759:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333182592/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019583.718767:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019583.718769:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019583.718773:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333182592:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019583.718778:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333182720:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019583.718792:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019583.718798:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019583/real 1734019583] req@ffff8800a73cf100 x1818247333182720/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019583.718808:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333182720/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019583.718815:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019583.718818:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019583.718821:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333182720:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019583.718845:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333182848:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019583.718857:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019583.718863:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019583/real 1734019583] req@ffff8800a73cc700 x1818247333182848/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019583.718881:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333182848/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019583.718888:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019583.718891:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019583.718893:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333182848:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019583.718898:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333182976:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019583.718910:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019583.718916:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019583/real 1734019583] req@ffff8800a73cdf80 x1818247333182976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019583.718926:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333182976/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019583.718933:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019583.718935:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019583.718938:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333182976:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019583.718943:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333183104:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019583.718956:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019583.718962:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019583/real 1734019583] req@ffff8800a73cf800 x1818247333183104/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019583.718972:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333183104/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019583.718979:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019583.718982:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019583.718985:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333183104:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019583.718990:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333183232:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019583.719002:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019583.719008:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019583/real 1734019583] req@ffff8800a73cfb80 x1818247333183232/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019583.719018:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333183232/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019583.719025:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019583.719027:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019583.719030:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333183232:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019583.868482:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019584.870438:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019585.872486:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019586.874504:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019587.876471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019588.726375:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019588.726388:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019588.726394:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019588.726407:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3851 00000100:00080000:3.0:1734019588.726419:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019588.726428:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019588.726477:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333183360/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019588.726514:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019588.726519:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019588.726521:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019588.726526:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3851 00000100:00080000:3.0:1734019588.726530:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019588.726534:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019588.726545:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019588.726560:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333183488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019588.726571:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019588.726574:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019588.726576:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019588.726580:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3851 00000100:00080000:3.0:1734019588.726583:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019588.726587:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019588.726592:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019588.726601:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333183616/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019588.726612:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019588.726615:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019588.726617:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734019588.726619:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333183360:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019588.726621:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3851 00000100:00080000:3.0:1734019588.726623:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019588.726626:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019588.726631:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019588.726639:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333183744/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019588.726647:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019588.726650:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019588.726652:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019588.726655:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3851 00000100:00080000:3.0:1734019588.726658:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019588.726662:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019588.726666:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019588.726675:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333183872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019588.726683:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019588.726686:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019588.726688:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019588.726693:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3851 00000100:00080000:3.0:1734019588.726697:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019588.726701:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734019588.726703:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734019588.726705:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019588.726714:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333184000/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019588.726722:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019588.726725:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019588.726727:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734019588.726729:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019588/real 1734019588] req@ffff8800a73cdc00 x1818247333183360/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019588.726730:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3851 00000100:00080000:3.0:1734019588.726733:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019588.726736:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019588.726741:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019588.726750:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333184128/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019588.726751:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333183360/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019588.726763:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019588.726766:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019588.726771:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333183360:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019588.726779:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333183488:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019588.726802:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019588.726809:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019588/real 1734019588] req@ffff8800a73cfb80 x1818247333183488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019588.726820:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333183488/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019588.726828:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019588.726831:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019588.726834:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333183488:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019588.726862:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333183616:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019588.726877:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019588.726883:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019588/real 1734019588] req@ffff8800a73cf800 x1818247333183616/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019588.726893:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333183616/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019588.726901:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019588.726903:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019588.726906:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333183616:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019588.726911:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333183744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019588.726923:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019588.726929:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019588/real 1734019588] req@ffff8800a73cdf80 x1818247333183744/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019588.726945:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333183744/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019588.726953:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019588.726955:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019588.726958:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333183744:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019588.726963:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333183872:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019588.726976:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019588.726982:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019588/real 1734019588] req@ffff8800a73cc700 x1818247333183872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019588.726991:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333183872/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019588.726999:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019588.727001:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019588.727003:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333183872:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019588.727008:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333184000:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019588.727070:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019588.727078:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019588/real 1734019588] req@ffff8800a73cf100 x1818247333184000/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019588.727089:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333184000/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019588.727096:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019588.727099:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019588.727103:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333184000:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019588.727109:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333184128:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019588.727122:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019588.727128:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019588/real 1734019588] req@ffff8800a73cc380 x1818247333184128/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019588.727138:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333184128/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019588.727146:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019588.727149:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019588.727152:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333184128:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019588.878505:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019589.880503:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019590.882520:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019591.884463:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019592.886417:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019593.734369:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019593.734378:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019593.734382:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019593.734393:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3856 00000100:00080000:3.0:1734019593.734403:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019593.734411:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019593.734449:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333184256/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019593.734481:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019593.734486:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019593.734488:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019593.734492:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3856 00000100:00080000:3.0:1734019593.734495:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019593.734499:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019593.734509:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019593.734519:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333184384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019593.734529:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019593.734532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019593.734534:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019593.734537:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3856 00000100:00080000:3.0:1734019593.734540:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019593.734544:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019593.734548:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019593.734556:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333184512/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019593.734565:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019593.734568:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019593.734569:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019593.734573:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3856 00000100:00080000:3.0:1734019593.734576:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019593.734579:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019593.734583:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019593.734591:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333184640/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019593.734594:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333184256:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019593.734600:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019593.734603:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019593.734605:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019593.734608:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3856 00000100:00080000:3.0:1734019593.734610:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019593.734613:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019593.734618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019593.734625:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333184768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019593.734633:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019593.734636:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019593.734637:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019593.734642:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3856 00000100:00080000:3.0:1734019593.734644:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019593.734648:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019593.734652:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019593.734659:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333184896/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734019593.734663:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019593.734668:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019593.734671:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019593.734673:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019593.734676:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3856 00000100:00080000:3.0:1734019593.734680:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019593.734684:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019593.734685:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019593/real 1734019593] req@ffff8800a73cf480 x1818247333184256/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000008:00100000:3.0:1734019593.734688:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019593.734697:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333185024/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019593.734705:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333184256/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019593.734715:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019593.734718:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019593.734723:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333184256:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019593.734730:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333184384:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019593.734751:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019593.734758:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019593/real 1734019593] req@ffff8800a73cea00 x1818247333184384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019593.734770:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333184384/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019593.734777:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019593.734780:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019593.734783:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333184384:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019593.734789:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333184512:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019593.734803:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019593.734809:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019593/real 1734019593] req@ffff8800a73cd880 x1818247333184512/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019593.734819:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333184512/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019593.734827:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019593.734829:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019593.734832:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333184512:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019593.734856:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333184640:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019593.734868:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019593.734874:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019593/real 1734019593] req@ffff8800a73ced80 x1818247333184640/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019593.734891:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333184640/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019593.734899:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019593.734901:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019593.734904:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333184640:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019593.734908:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333184768:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019593.734920:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019593.734926:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019593/real 1734019593] req@ffff8800a73cce00 x1818247333184768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019593.734979:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333184768/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019593.734987:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019593.734989:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019593.734992:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333184768:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019593.734998:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333184896:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019593.735013:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019593.735020:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019593/real 1734019593] req@ffff8800a73ce680 x1818247333184896/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019593.735030:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333184896/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019593.735038:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019593.735041:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019593.735044:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333184896:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019593.735049:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333185024:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019593.735062:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019593.735068:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019593/real 1734019593] req@ffff8800a73cd500 x1818247333185024/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019593.735077:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333185024/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019593.735085:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019593.735087:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019593.735090:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333185024:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019593.888501:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019594.890465:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019595.892442:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019596.894460:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019597.896424:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019598.742370:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019598.742379:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019598.742383:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019598.742394:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3861 00000100:00080000:3.0:1734019598.742404:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019598.742412:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019598.742449:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333185152/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019598.742496:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019598.742501:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019598.742503:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019598.742508:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3861 00000100:00080000:3.0:1734019598.742511:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019598.742515:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019598.742531:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019598.742545:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333185280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019598.742556:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019598.742559:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019598.742561:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019598.742564:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3861 00000100:00080000:3.0:1734019598.742567:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019598.742571:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019598.742575:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019598.742584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333185408/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019598.742592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019598.742595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019598.742596:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019598.742600:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3861 00000100:00080000:3.0:1734019598.742602:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019598.742606:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019598.742610:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019598.742618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333185536/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019598.742621:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333185152:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019598.742627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019598.742631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019598.742633:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019598.742636:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3861 00000100:00080000:3.0:1734019598.742638:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019598.742642:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019598.742646:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019598.742653:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333185664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019598.742661:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019598.742664:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019598.742666:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019598.742670:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3861 00000100:00080000:3.0:1734019598.742672:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019598.742676:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019598.742680:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019598.742687:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333185792/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019598.742695:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019598.742698:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019598.742700:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1734019598.742702:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019598.742703:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3861 00000100:00080000:3.0:1734019598.742706:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019598.742709:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019598.742714:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019598.742721:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333185920/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019598.742732:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019598/real 1734019598] req@ffff8800a73cca80 x1818247333185152/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019598.742759:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333185152/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019598.742775:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019598.742780:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019598.742787:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333185152:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019598.742798:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333185280:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019598.742831:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019598.742840:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019598/real 1734019598] req@ffff8800a73cd500 x1818247333185280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019598.742885:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333185280/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019598.742894:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019598.742896:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019598.742900:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333185280:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019598.742906:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333185408:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019598.742923:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019598.742929:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019598/real 1734019598] req@ffff8800a73ce680 x1818247333185408/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019598.742940:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333185408/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019598.742948:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019598.742950:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019598.742953:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333185408:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019598.742976:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333185536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019598.742988:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019598.742994:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019598/real 1734019598] req@ffff8800a73cce00 x1818247333185536/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019598.743011:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333185536/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019598.743018:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019598.743020:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019598.743023:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333185536:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019598.743028:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333185664:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019598.743040:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019598.743046:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019598/real 1734019598] req@ffff8800a73ced80 x1818247333185664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019598.743055:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333185664/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019598.743063:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019598.743064:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019598.743067:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333185664:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019598.743072:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333185792:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019598.743084:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019598.743090:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019598/real 1734019598] req@ffff8800a73cd880 x1818247333185792/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019598.743100:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333185792/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019598.743107:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019598.743110:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019598.743113:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333185792:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019598.743119:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333185920:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019598.743131:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019598.743136:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019598/real 1734019598] req@ffff8800a73cea00 x1818247333185920/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019598.743146:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333185920/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019598.743153:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019598.743155:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019598.743158:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333185920:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019598.898487:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019599.900457:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019600.902478:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019601.904497:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019602.906486:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019603.750334:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019603.750343:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019603.750347:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019603.750357:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3866 00000100:00080000:3.0:1734019603.750366:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019603.750374:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019603.750407:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333186048/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019603.750441:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019603.750446:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019603.750447:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019603.750452:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3866 00000100:00080000:3.0:1734019603.750455:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019603.750459:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019603.750468:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019603.750479:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333186176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019603.750489:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019603.750493:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019603.750495:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019603.750498:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3866 00000100:00080000:3.0:1734019603.750501:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019603.750504:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019603.750509:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019603.750517:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333186304/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019603.750525:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019603.750528:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019603.750530:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019603.750533:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3866 00000100:00080000:3.0:1734019603.750535:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019603.750539:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019603.750543:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019603.750549:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333186048:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019603.750551:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333186432/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019603.750559:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019603.750563:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019603.750564:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019603.750568:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3866 00000100:00080000:3.0:1734019603.750570:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019603.750574:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019603.750578:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019603.750586:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333186560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019603.750594:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019603.750596:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019603.750598:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019603.750602:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3866 00000100:00080000:3.0:1734019603.750604:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019603.750608:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019603.750612:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734019603.750614:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019603.750619:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333186688/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019603.750627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019603.750630:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019603.750631:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019603.750635:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3866 00000100:00100000:0.0:1734019603.750637:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019603/real 1734019603] req@ffff8800a73cf480 x1818247333186048/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019603.750638:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019603.750641:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019603.750645:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019603.750654:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333186816/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019603.750657:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333186048/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019603.750667:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019603.750671:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019603.750676:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333186048:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019603.750683:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333186176:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019603.750703:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019603.750710:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019603/real 1734019603] req@ffff8800a73cc380 x1818247333186176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019603.750721:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333186176/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019603.750729:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019603.750732:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019603.750735:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333186176:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019603.750741:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333186304:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019603.750754:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019603.750806:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019603/real 1734019603] req@ffff8800a73cf100 x1818247333186304/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019603.750819:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333186304/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019603.750827:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019603.750829:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019603.750832:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333186304:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019603.750856:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333186432:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019603.750873:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019603.750880:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019603/real 1734019603] req@ffff8800a73cc700 x1818247333186432/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019603.750898:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333186432/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019603.750906:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019603.750908:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019603.750911:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333186432:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019603.750916:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333186560:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019603.750927:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019603.750933:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019603/real 1734019603] req@ffff8800a73cdf80 x1818247333186560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019603.750943:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333186560/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019603.750950:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019603.750952:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019603.750955:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333186560:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019603.750960:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333186688:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019603.750972:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019603.750978:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019603/real 1734019603] req@ffff8800a73cf800 x1818247333186688/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019603.750988:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333186688/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019603.750995:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019603.750998:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019603.751001:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333186688:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019603.751007:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333186816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019603.751019:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019603.751025:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019603/real 1734019603] req@ffff8800a73cfb80 x1818247333186816/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019603.751034:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333186816/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019603.751042:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019603.751044:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019603.751046:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333186816:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019603.908470:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019604.910483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019605.912487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019606.914461:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019607.916492:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019608.758333:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019608.758342:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019608.758347:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019608.758357:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3871 00000100:00080000:3.0:1734019608.758367:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019608.758374:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019608.758413:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333186944/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019608.758446:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019608.758450:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019608.758452:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019608.758456:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3871 00000100:00080000:3.0:1734019608.758459:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019608.758463:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019608.758473:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019608.758485:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333187072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019608.758495:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019608.758499:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019608.758501:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019608.758505:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3871 00000100:00080000:3.0:1734019608.758507:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019608.758511:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019608.758516:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019608.758524:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333187200/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019608.758532:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019608.758535:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019608.758537:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019608.758540:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3871 00000100:00080000:3.0:1734019608.758542:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019608.758546:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019608.758550:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333186944:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734019608.758550:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019608.758558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333187328/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019608.758566:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019608.758570:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019608.758571:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019608.758574:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3871 00000100:00080000:3.0:1734019608.758577:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019608.758580:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019608.758585:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019608.758592:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333187456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019608.758601:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019608.758604:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019608.758606:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019608.758610:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3871 00000100:00080000:3.0:1734019608.758612:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734019608.758615:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019608.758616:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019608.758620:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019608.758628:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333187584/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019608.758636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734019608.758637:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019608/real 1734019608] req@ffff8800a73cdc00 x1818247333186944/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019608.758639:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019608.758641:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019608.758644:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3871 00000100:00080000:3.0:1734019608.758647:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019608.758650:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019608.758654:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019608.758656:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333186944/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019608.758662:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333187712/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019608.758666:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019608.758670:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019608.758708:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333186944:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019608.758717:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333187072:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019608.758742:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019608.758751:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019608/real 1734019608] req@ffff8800a73cfb80 x1818247333187072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019608.758762:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333187072/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019608.758770:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019608.758773:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019608.758776:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333187072:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019608.758782:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333187200:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019608.758796:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019608.758802:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019608/real 1734019608] req@ffff8800a73cf800 x1818247333187200/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019608.758812:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333187200/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019608.758819:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019608.758822:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019608.758825:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333187200:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019608.758848:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333187328:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019608.758860:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019608.758867:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019608/real 1734019608] req@ffff8800a73cdf80 x1818247333187328/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019608.758884:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333187328/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019608.758892:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019608.758894:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019608.758897:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333187328:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019608.758903:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333187456:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019608.758914:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019608.758920:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019608/real 1734019608] req@ffff8800a73cc700 x1818247333187456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019608.758930:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333187456/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019608.758937:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019608.758940:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019608.758942:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333187456:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019608.758948:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333187584:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019608.758959:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019608.758965:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019608/real 1734019608] req@ffff8800a73cf100 x1818247333187584/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019608.758974:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333187584/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019608.758981:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019608.758985:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019608.758988:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333187584:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019608.758994:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333187712:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019608.759006:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019608.759011:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019608/real 1734019608] req@ffff8800a73cc380 x1818247333187712/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019608.759021:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333187712/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019608.759028:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019608.759030:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019608.759033:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333187712:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019608.918468:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019609.920476:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019610.922409:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019611.924406:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019612.926366:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019613.766371:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019613.766387:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019613.766392:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019613.766406:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3876 00000100:00080000:3.0:1734019613.766419:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019613.766428:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019613.766477:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333187840/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019613.766514:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019613.766519:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019613.766521:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019613.766526:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3876 00000100:00080000:3.0:1734019613.766530:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019613.766534:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019613.766546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019613.766558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333187968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019613.766569:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019613.766572:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019613.766574:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019613.766579:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3876 00000100:00080000:3.0:1734019613.766581:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019613.766585:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019613.766627:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019613.766639:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333188096/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019613.766645:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333187840:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019613.766650:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019613.766653:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019613.766655:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019613.766659:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3876 00000100:00080000:3.0:1734019613.766662:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019613.766666:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019613.766670:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019613.766679:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333188224/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019613.766687:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019613.766691:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019613.766692:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019613.766696:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3876 00000100:00080000:3.0:1734019613.766699:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019613.766703:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019613.766707:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019613.766716:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333188352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019613.766725:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1734019613.766727:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019613.766728:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019613.766730:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019613.766734:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3876 00000100:00080000:3.0:1734019613.766738:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019613.766742:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019613.766746:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019613.766752:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019613/real 1734019613] req@ffff8800a73cf480 x1818247333187840/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019613.766755:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333188480/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019613.766763:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019613.766766:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019613.766768:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019613.766771:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3876 00000100:00080000:3.0:1734019613.766774:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734019613.766775:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333187840/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019613.766777:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019613.766782:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1734019613.766787:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019613.766790:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734019613.766792:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333188608/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019613.766796:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333187840:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019613.766803:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333187968:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019613.766826:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019613.766833:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019613/real 1734019613] req@ffff8800a73cea00 x1818247333187968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019613.766844:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333187968/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019613.766852:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019613.766855:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019613.766858:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333187968:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019613.766885:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333188096:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019613.766900:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019613.766906:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019613/real 1734019613] req@ffff8800a73cd880 x1818247333188096/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019613.766916:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333188096/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019613.766924:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019613.766926:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019613.766930:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333188096:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019613.766936:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333188224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019613.766948:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019613.766954:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019613/real 1734019613] req@ffff8800a73ced80 x1818247333188224/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019613.766971:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333188224/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019613.766979:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019613.766981:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019613.766984:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333188224:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019613.766988:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333188352:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019613.767001:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019613.767007:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019613/real 1734019613] req@ffff8800a73cce00 x1818247333188352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019613.767017:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333188352/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019613.767024:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019613.767026:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019613.767029:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333188352:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019613.767034:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333188480:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019613.767047:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019613.767053:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019613/real 1734019613] req@ffff8800a73ce680 x1818247333188480/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019613.767064:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333188480/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019613.767071:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019613.767074:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019613.767077:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333188480:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019613.767082:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333188608:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019613.767095:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019613.767101:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019613/real 1734019613] req@ffff8800a73cd500 x1818247333188608/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019613.767110:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333188608/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019613.767118:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019613.767120:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019613.767123:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333188608:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734019613.928398:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019614.930396:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019615.932416:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019616.934365:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019617.936504:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019618.774366:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019618.774378:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019618.774384:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019618.774398:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3881 00000100:00080000:3.0:1734019618.774409:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019618.774420:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019618.774470:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333188736/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019618.774538:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019618.774543:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019618.774545:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019618.774550:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3881 00000100:00080000:3.0:1734019618.774554:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019618.774558:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019618.774571:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019618.774587:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333188864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019618.774598:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019618.774601:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019618.774603:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019618.774607:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3881 00000100:00080000:3.0:1734019618.774610:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019618.774613:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019618.774618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:0.0:1734019618.774624:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333188736:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019618.774627:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333188992/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019618.774636:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019618.774640:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019618.774641:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019618.774645:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3881 00000100:00080000:3.0:1734019618.774647:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019618.774651:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019618.774656:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019618.774664:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333189120/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019618.774673:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019618.774676:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019618.774677:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019618.774681:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3881 00000100:00080000:3.0:1734019618.774684:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019618.774688:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019618.774692:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019618.774701:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333189248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019618.774709:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019618.774712:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019618.774713:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019618.774718:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3881 00000100:00080000:3.0:1734019618.774721:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019618.774724:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019618.774729:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019618.774736:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333189376/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019618.774738:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333188864:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734019618.774744:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019618.774749:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019618.774750:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019618.774754:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3881 00000100:00080000:3.0:1734019618.774758:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019618.774761:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019618.774766:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019618.774774:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333188992:192.168.201.152@tcp:8:kworker.0 00000100:00100000:3.0:1734019618.774774:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333189504/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019618.774792:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333189120:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019618.774807:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333189248:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019618.774822:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333189376:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019618.774839:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333189504:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019618.938496:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019619.940481:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019620.942537:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1734019621.780545:0:1957:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:1.0:1734019621.780596:0:1957:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1734019621.780617:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019621.780654:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019621.780662:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019621.780669:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019621.780676:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019621.780682:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019621.780688:0:1957:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:0.0:1734019621.780804:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019618/real 1734019621] req@ffff8800a73cea00 x1818247333189504/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019621.780854:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333189504/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019621.780874:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019621.780881:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019621.780890:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333189504:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019621.780903:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019618/real 1734019621] req@ffff8800a73ce680 x1818247333188992/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019621.780926:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333188992/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019621.780939:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019621.780944:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019621.780950:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333188992:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019621.780962:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019618/real 1734019621] req@ffff8800a73cce00 x1818247333189120/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019621.780980:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333189120/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019621.780993:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019621.780998:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019621.781003:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333189120:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019621.781015:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019618/real 1734019621] req@ffff8800a73ced80 x1818247333189248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019621.781048:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333189248/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019621.781061:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019621.781065:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019621.781071:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333189248:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019621.781082:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019618/real 1734019621] req@ffff8800a73cd880 x1818247333189376/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019621.781102:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333189376/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019621.781115:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019621.781119:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019621.781125:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333189376:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019621.781136:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019618/real 1734019621] req@ffff8800a73cca80 x1818247333188736/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019621.781157:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333188736/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019621.781170:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019621.781174:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019621.781180:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333188736:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019621.781192:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019618/real 1734019621] req@ffff8800a73cd500 x1818247333188864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019621.781279:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333188864/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019621.781305:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019621.781310:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019621.781317:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333188864:192.168.201.152@tcp:38:kworker.0 00010000:00010000:2.0:1734019621.944546:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019622.946521:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019623.782337:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019623.782346:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019623.782351:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019623.782362:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3886 00000100:00080000:3.0:1734019623.782370:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019623.782378:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019623.782415:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333189632/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019623.782489:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019623.782496:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019623.782498:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019623.782505:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3886 00000100:00080000:3.0:1734019623.782510:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019623.782517:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019623.782531:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019623.782550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333189760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019623.782563:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333189632:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019623.782567:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019623.782572:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019623.782574:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019623.782578:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3886 00000100:00080000:3.0:1734019623.782583:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019623.782587:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019623.782594:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019623.782606:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333189888/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019623.782618:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019623.782622:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019623.782624:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019623.782628:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3886 00000100:00080000:3.0:1734019623.782632:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734019623.782635:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019623.782638:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019623.782644:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019623.782655:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333190016/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019623.782659:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019623/real 1734019623] req@ffff8800a73cf480 x1818247333189632/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019623.782668:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019623.782673:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019623.782676:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019623.782681:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3886 00000100:00100000:0.0:1734019623.782683:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333189632/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019623.782685:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019623.782691:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:0.0:1734019623.782697:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1734019623.782697:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1734019623.782702:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019623.782709:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333189632:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019623.782709:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333190144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019623.782722:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019623.782727:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019623.782729:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019623.782735:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3886 00000100:00100000:0.0:1734019623.782736:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333189760:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734019623.782739:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019623.782744:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019623.782750:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1734019623.782763:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019623.782763:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333190272/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019623.782774:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019623/real 1734019623] req@ffff8800a73cc380 x1818247333189760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019623.782777:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019623.782783:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019623.782786:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019623.782791:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3886 00000100:00100000:0.0:1734019623.782792:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333189760/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019623.782795:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019623.782801:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:0.0:1734019623.782805:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1734019623.782807:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1734019623.782809:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019623.782814:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333189760:192.168.201.152@tcp:38:kworker.0 00000100:00100000:3.0:1734019623.782820:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333190400/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019623.782823:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333189888:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019623.782846:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019623.782856:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019623/real 1734019623] req@ffff8800a73cf100 x1818247333189888/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019623.782874:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333189888/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019623.782887:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019623.782890:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019623.782896:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333189888:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019623.782905:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333190016:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019623.782931:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019623.782941:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019623/real 1734019623] req@ffff8800a73cc700 x1818247333190016/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019623.782953:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333190016/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019623.782961:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019623.782963:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019623.782966:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333190016:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019623.782971:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333190144:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019623.782990:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019623.782996:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019623/real 1734019623] req@ffff8800a73cdf80 x1818247333190144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019623.783007:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333190144/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019623.783014:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019623.783016:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019623.783019:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333190144:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019623.783036:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333190272:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019623.783048:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019623.783054:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019623/real 1734019623] req@ffff8800a73cf800 x1818247333190272/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019623.783064:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333190272/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019623.783071:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019623.783073:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019623.783077:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333190272:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019623.783082:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333190400:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019623.783093:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019623.783099:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019623/real 1734019623] req@ffff8800a73cfb80 x1818247333190400/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019623.783115:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333190400/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019623.783123:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019623.783125:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019623.783128:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333190400:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734019623.948510:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019624.950456:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019625.952468:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019626.954426:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019627.956408:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019628.790370:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019628.790377:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019628.790380:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019628.790386:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3891 00000100:00080000:3.0:1734019628.790392:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019628.790397:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019628.790420:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333190528/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019628.790442:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019628.790444:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019628.790445:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019628.790462:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3891 00000100:00080000:3.0:1734019628.790463:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019628.790465:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019628.790471:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019628.790478:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333190656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019628.790482:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019628.790484:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019628.790485:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019628.790486:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3891 00000100:00080000:3.0:1734019628.790488:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019628.790490:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019628.790492:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019628.790496:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333190784/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019628.790499:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019628.790500:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019628.790501:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019628.790503:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3891 00000100:00080000:3.0:1734019628.790504:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019628.790505:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019628.790507:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019628.790511:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333190912/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019628.790514:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019628.790515:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019628.790516:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019628.790518:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3891 00000100:00080000:3.0:1734019628.790519:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019628.790520:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019628.790522:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019628.790526:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333190528:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019628.790526:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333191040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019628.790532:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019628.790533:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019628.790534:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019628.790536:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3891 00000100:00080000:3.0:1734019628.790537:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019628.790538:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019628.790540:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019628.790543:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333191168/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019628.790546:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019628.790548:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019628.790548:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019628.790550:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3891 00000100:00080000:3.0:1734019628.790551:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019628.790552:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019628.790554:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019628.790558:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333191296/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734019628.790564:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019628.790577:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019628/real 1734019628] req@ffff8800a73cdc00 x1818247333190528/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019628.790586:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333190528/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019628.790590:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019628.790592:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019628.790594:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333190528:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019628.790597:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333190656:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019628.790605:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019628.790608:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019628/real 1734019628] req@ffff8800a73cfb80 x1818247333190656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019628.790612:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333190656/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019628.790616:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019628.790617:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019628.790619:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333190656:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019628.790622:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333190784:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019628.790628:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019628.790630:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019628/real 1734019628] req@ffff8800a73cf800 x1818247333190784/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019628.790634:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333190784/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019628.790638:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019628.790639:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019628.790640:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333190784:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019628.790643:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333190912:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019628.790648:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019628.790650:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019628/real 1734019628] req@ffff8800a73cdf80 x1818247333190912/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019628.790654:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333190912/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019628.790657:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019628.790658:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019628.790659:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333190912:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019628.790672:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333191040:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019628.790678:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019628.790681:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019628/real 1734019628] req@ffff8800a73cc700 x1818247333191040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019628.790686:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333191040/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019628.790689:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019628.790690:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019628.790691:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333191040:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019628.790693:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333191168:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019628.790698:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019628.790700:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019628/real 1734019628] req@ffff8800a73cf100 x1818247333191168/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019628.790704:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333191168/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019628.790707:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019628.790708:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019628.790710:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333191168:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019628.790712:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333191296:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019628.790717:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019628.790720:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019628/real 1734019628] req@ffff8800a73cc380 x1818247333191296/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019628.790727:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333191296/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019628.790730:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019628.790731:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019628.790732:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333191296:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019628.958445:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019629.960445:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019630.962415:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019631.964390:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019632.966433:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019633.798369:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019633.798376:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019633.798379:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019633.798387:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3896 00000100:00080000:3.0:1734019633.798394:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019633.798399:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019633.798425:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333191424/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019633.798451:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019633.798454:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019633.798455:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019633.798459:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3896 00000100:00080000:3.0:1734019633.798462:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019633.798465:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019633.798473:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019633.798483:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333191552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019633.798491:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019633.798494:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019633.798495:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019633.798498:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3896 00000100:00080000:3.0:1734019633.798499:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019633.798502:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019633.798506:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333191424:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734019633.798506:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019633.798512:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333191680/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019633.798523:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019633.798525:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019633.798526:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019633.798529:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3896 00000100:00080000:3.0:1734019633.798530:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019633.798533:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019633.798536:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019633.798542:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333191808/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019633.798548:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019633.798550:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019633.798551:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1734019633.798553:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019633.798554:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3896 00000100:00080000:3.0:1734019633.798556:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019633.798558:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019633.798562:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019633.798568:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019633/real 1734019633] req@ffff8800a73cf480 x1818247333191424/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019633.798568:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333191936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019633.798575:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019633.798578:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019633.798579:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734019633.798581:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333191424/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019633.798582:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3896 00000100:00080000:3.0:1734019633.798585:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019633.798587:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00080000:0.0:1734019633.798589:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1734019633.798590:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1734019633.798591:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019633.798594:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333191424:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019633.798596:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333192064/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019633.798600:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333191552:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734019633.798603:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019633.798605:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019633.798606:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019633.798609:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3896 00000100:00080000:3.0:1734019633.798612:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734019633.798613:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019633.798614:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019633.798618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019633.798619:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019633/real 1734019633] req@ffff8800a73cd500 x1818247333191552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019633.798625:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333192192/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019633.798628:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333191552/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019633.798634:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019633.798637:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019633.798640:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333191552:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019633.798659:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333191680:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019633.798673:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019633.798678:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019633/real 1734019633] req@ffff8800a73cca80 x1818247333191680/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019633.798687:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333191680/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019633.798693:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019633.798695:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019633.798698:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333191680:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019633.798702:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333191808:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019633.798713:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019633.798717:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019633/real 1734019633] req@ffff8800a73cd880 x1818247333191808/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019633.798722:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333191808/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019633.798727:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019633.798728:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019633.798729:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333191808:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019633.798732:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333191936:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019633.798740:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019633.798743:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019633/real 1734019633] req@ffff8800a73ced80 x1818247333191936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019633.798749:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333191936/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019633.798752:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019633.798753:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019633.798755:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333191936:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019633.798757:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333192064:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019633.798764:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019633.798767:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019633/real 1734019633] req@ffff8800a73cce00 x1818247333192064/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019633.798771:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333192064/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019633.798775:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019633.798776:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019633.798777:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333192064:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019633.798780:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333192192:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019633.798786:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019633.798789:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019633/real 1734019633] req@ffff8800a73ce680 x1818247333192192/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019633.798798:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333192192/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019633.798802:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019633.798803:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019633.798805:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333192192:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734019633.968446:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019634.970458:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019635.972489:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019636.974396:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019637.976425:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019638.806316:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019638.806325:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019638.806330:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019638.806341:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3901 00000100:00080000:3.0:1734019638.806350:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019638.806358:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019638.806395:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333192320/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019638.806436:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019638.806440:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019638.806442:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019638.806447:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3901 00000100:00080000:3.0:1734019638.806450:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019638.806454:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019638.806464:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019638.806476:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333192448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019638.806486:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019638.806489:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019638.806491:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019638.806495:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3901 00000100:00080000:3.0:1734019638.806497:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019638.806501:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019638.806505:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019638.806513:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333192576/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019638.806522:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019638.806525:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019638.806526:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019638.806530:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3901 00000100:00080000:3.0:1734019638.806532:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019638.806537:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019638.806541:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019638.806542:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333192320:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019638.806549:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333192704/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019638.806558:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019638.806561:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019638.806562:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019638.806566:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3901 00000100:00080000:3.0:1734019638.806568:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019638.806571:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019638.806576:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019638.806584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333192832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019638.806592:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019638.806595:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019638.806596:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019638.806600:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3901 00000100:00080000:3.0:1734019638.806603:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019638.806607:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734019638.806609:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734019638.806611:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019638.806618:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333192960/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019638.806627:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734019638.806630:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019638/real 1734019638] req@ffff8800a73cea00 x1818247333192320/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019638.806631:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019638.806632:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019638.806636:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3901 00000100:00080000:3.0:1734019638.806639:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019638.806642:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019638.806647:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019638.806649:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333192320/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019638.806655:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333193088/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019638.806660:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019638.806663:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019638.806668:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333192320:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019638.806675:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333192448:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019638.806693:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019638.806699:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019638/real 1734019638] req@ffff8800a73ce680 x1818247333192448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019638.806710:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333192448/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019638.806718:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019638.806721:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019638.806725:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333192448:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019638.806731:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333192576:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019638.806746:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019638.806751:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019638/real 1734019638] req@ffff8800a73cce00 x1818247333192576/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019638.806761:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333192576/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019638.806769:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019638.806771:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019638.806775:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333192576:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019638.806799:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333192704:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019638.806811:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019638.806817:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019638/real 1734019638] req@ffff8800a73ced80 x1818247333192704/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019638.806827:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333192704/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019638.806835:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019638.806837:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019638.806840:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333192704:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019638.806846:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333192832:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019638.806861:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019638.806867:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019638/real 1734019638] req@ffff8800a73cd880 x1818247333192832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019638.806878:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333192832/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019638.806886:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019638.806888:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019638.806891:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333192832:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019638.806897:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333192960:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019638.806908:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019638.806914:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019638/real 1734019638] req@ffff8800a73cca80 x1818247333192960/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019638.806924:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333192960/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019638.806931:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019638.806933:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019638.806936:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333192960:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019638.806941:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333193088:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019638.806953:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019638.806959:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019638/real 1734019638] req@ffff8800a73cd500 x1818247333193088/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019638.806976:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333193088/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019638.806984:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019638.806986:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019638.806989:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333193088:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019638.978483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019639.980473:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019640.982446:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019641.984494:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019642.986407:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019643.814338:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019643.814347:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019643.814350:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019643.814359:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3906 00000100:00080000:3.0:1734019643.814368:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019643.814375:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019643.814419:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333193216/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019643.814442:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019643.814444:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019643.814446:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019643.814450:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3906 00000100:00080000:3.0:1734019643.814452:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019643.814455:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019643.814464:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019643.814473:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333193344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019643.814480:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019643.814482:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019643.814484:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734019643.814486:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333193216:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019643.814486:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3906 00000100:00080000:3.0:1734019643.814488:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019643.814491:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019643.814494:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019643.814500:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333193472/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019643.814506:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019643.814507:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019643.814508:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019643.814511:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3906 00000100:00080000:3.0:1734019643.814513:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019643.814515:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019643.814518:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019643.814524:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333193600/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019643.814529:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019643.814531:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019643.814532:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019643.814535:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3906 00000100:00080000:3.0:1734019643.814536:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019643.814538:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734019643.814540:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734019643.814541:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019643.814547:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333193728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019643.814553:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019643.814555:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019643.814556:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1734019643.814559:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019643/real 1734019643] req@ffff8800a73cf480 x1818247333193216/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019643.814559:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3906 00000100:00080000:3.0:1734019643.814561:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019643.814563:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019643.814566:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019643.814571:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333193216/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019643.814572:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333193856/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019643.814578:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1734019643.814579:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1734019643.814580:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019643.814581:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1734019643.814582:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00080000:3.0:1734019643.814584:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3906 00000100:00100000:0.0:1734019643.814585:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333193216:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019643.814585:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019643.814587:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019643.814590:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019643.814596:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333193984/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019643.814602:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333193344:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019643.814616:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019643.814620:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019643/real 1734019643] req@ffff8800a73cc380 x1818247333193344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019643.814627:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333193344/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019643.814633:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019643.814635:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019643.814637:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333193344:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019643.814641:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333193472:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019643.814654:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019643.814659:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019643/real 1734019643] req@ffff8800a73cf100 x1818247333193472/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019643.814664:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333193472/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019643.814667:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019643.814668:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019643.814670:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333193472:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019643.814672:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333193600:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019643.814679:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019643.814682:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019643/real 1734019643] req@ffff8800a73cc700 x1818247333193600/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019643.814686:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333193600/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019643.814690:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019643.814691:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019643.814692:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333193600:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019643.814694:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333193728:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019643.814701:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019643.814704:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019643/real 1734019643] req@ffff8800a73cdf80 x1818247333193728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019643.814708:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333193728/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019643.814712:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019643.814713:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019643.814714:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333193728:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019643.814716:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333193856:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019643.814721:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019643.814723:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019643/real 1734019643] req@ffff8800a73cf800 x1818247333193856/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019643.814727:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333193856/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019643.814730:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019643.814731:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019643.814732:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333193856:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019643.814741:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333193984:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019643.814747:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019643.814749:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019643/real 1734019643] req@ffff8800a73cfb80 x1818247333193984/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019643.814757:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333193984/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019643.814761:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019643.814761:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019643.814763:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333193984:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734019643.988471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019644.990331:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019645.992390:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019646.994475:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019647.996418:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019648.822356:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019648.822362:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019648.822364:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019648.822371:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3911 00000100:00080000:3.0:1734019648.822377:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019648.822382:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019648.822405:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333194112/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019648.822427:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019648.822429:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019648.822430:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019648.822432:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3911 00000100:00080000:3.0:1734019648.822433:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019648.822435:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019648.822441:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019648.822448:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333194240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019648.822453:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019648.822455:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019648.822456:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019648.822458:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3911 00000100:00080000:3.0:1734019648.822460:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019648.822462:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019648.822464:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019648.822468:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333194368/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019648.822472:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019648.822473:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019648.822474:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019648.822475:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3911 00000100:00080000:3.0:1734019648.822477:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019648.822478:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019648.822480:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019648.822483:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333194496/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019648.822488:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019648.822489:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019648.822490:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019648.822492:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3911 00000100:00080000:3.0:1734019648.822493:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019648.822494:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019648.822496:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333194112:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734019648.822496:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019648.822500:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333194624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019648.822507:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019648.822508:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019648.822509:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019648.822511:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3911 00000100:00080000:3.0:1734019648.822512:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019648.822514:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019648.822516:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019648.822520:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333194752/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019648.822523:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019648.822524:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019648.822525:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019648.822527:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3911 00000100:00080000:3.0:1734019648.822528:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019648.822529:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019648.822531:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019648.822535:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333194880/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734019648.822566:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019648.822588:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019648/real 1734019648] req@ffff8800a73cdc00 x1818247333194112/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019648.822602:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333194112/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019648.822612:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019648.822615:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019648.822619:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333194112:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019648.822626:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333194240:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019648.822641:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019648.822646:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019648/real 1734019648] req@ffff8800a73cfb80 x1818247333194240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019648.822655:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333194240/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019648.822661:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019648.822664:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019648.822667:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333194240:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019648.822672:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333194368:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019648.822685:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019648.822689:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019648/real 1734019648] req@ffff8800a73cf800 x1818247333194368/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019648.822698:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333194368/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019648.822704:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019648.822706:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019648.822708:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333194368:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019648.822733:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333194496:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019648.822743:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019648.822748:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019648/real 1734019648] req@ffff8800a73cdf80 x1818247333194496/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019648.822757:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333194496/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019648.822764:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019648.822765:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019648.822768:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333194496:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019648.822772:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333194624:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019648.822785:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019648.822790:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019648/real 1734019648] req@ffff8800a73cc700 x1818247333194624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019648.822799:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333194624/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019648.822806:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019648.822808:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019648.822810:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333194624:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019648.822815:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333194752:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019648.822825:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019648.822830:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019648/real 1734019648] req@ffff8800a73cf100 x1818247333194752/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019648.822839:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333194752/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019648.822846:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019648.822848:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019648.822850:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333194752:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019648.822862:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333194880:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019648.822873:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019648.822878:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019648/real 1734019648] req@ffff8800a73cc380 x1818247333194880/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019648.822891:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333194880/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019648.822898:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019648.822900:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019648.822902:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333194880:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019648.998500:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019650.000366:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019651.002354:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019652.004453:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019653.006415:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019653.830324:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019653.830330:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019653.830333:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019653.830340:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3916 00000100:00080000:3.0:1734019653.830346:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019653.830351:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019653.830377:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333195008/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019653.830398:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019653.830400:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019653.830401:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019653.830418:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3916 00000100:00080000:3.0:1734019653.830419:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019653.830421:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019653.830428:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019653.830433:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333195136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019653.830438:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019653.830439:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019653.830440:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019653.830442:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3916 00000100:00080000:3.0:1734019653.830443:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019653.830445:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019653.830447:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019653.830451:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333195264/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019653.830458:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019653.830459:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019653.830460:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019653.830461:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3916 00000100:00080000:3.0:1734019653.830462:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019653.830464:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019653.830466:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019653.830469:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333195392/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019653.830473:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019653.830474:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019653.830475:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019653.830476:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3916 00000100:00080000:3.0:1734019653.830477:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019653.830479:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019653.830481:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1734019653.830482:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333195008:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019653.830484:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333195520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019653.830488:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019653.830490:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019653.830490:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019653.830492:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3916 00000100:00080000:3.0:1734019653.830493:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019653.830495:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019653.830497:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019653.830501:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333195648/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019653.830504:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019653.830505:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019653.830506:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019653.830508:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3916 00000100:00080000:3.0:1734019653.830509:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019653.830510:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019653.830512:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019653.830516:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333195776/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000800:00000100:0.0:1734019653.830524:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019653.830537:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019653/real 1734019653] req@ffff8800a73cf480 x1818247333195008/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019653.830546:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333195008/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019653.830552:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019653.830553:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019653.830555:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333195008:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019653.830558:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333195136:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019653.830566:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019653.830569:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019653/real 1734019653] req@ffff8800a73cd500 x1818247333195136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019653.830574:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333195136/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019653.830577:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019653.830579:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019653.830580:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333195136:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019653.830583:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333195264:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019653.830590:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019653.830592:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019653/real 1734019653] req@ffff8800a73cca80 x1818247333195264/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019653.830596:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333195264/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019653.830600:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019653.830601:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019653.830603:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333195264:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019653.830605:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333195392:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019653.830610:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019653.830612:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019653/real 1734019653] req@ffff8800a73cd880 x1818247333195392/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019653.830616:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333195392/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019653.830619:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019653.830620:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019653.830621:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333195392:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019653.830635:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333195520:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019653.830642:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019653.830645:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019653/real 1734019653] req@ffff8800a73ced80 x1818247333195520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019653.830650:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333195520/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019653.830653:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019653.830654:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019653.830655:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333195520:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019653.830657:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333195648:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019653.830662:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019653.830664:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019653/real 1734019653] req@ffff8800a73cce00 x1818247333195648/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019653.830668:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333195648/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019653.830671:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019653.830672:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019653.830674:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333195648:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019653.830676:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333195776:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019653.830681:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019653.830684:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019653/real 1734019653] req@ffff8800a73ce680 x1818247333195776/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019653.830691:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333195776/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019653.830694:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019653.830695:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019653.830696:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333195776:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734019654.008419:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019655.010402:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019656.012384:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019657.014405:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019658.016495:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019658.838336:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019658.838345:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019658.838350:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019658.838361:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3921 00000100:00080000:3.0:1734019658.838370:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019658.838377:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019658.838412:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333195904/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019658.838452:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019658.838457:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019658.838459:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019658.838463:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3921 00000100:00080000:3.0:1734019658.838466:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019658.838470:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019658.838480:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019658.838492:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333196032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019658.838502:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019658.838505:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019658.838507:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019658.838511:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3921 00000100:00080000:3.0:1734019658.838513:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019658.838517:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019658.838521:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019658.838529:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333196160/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019658.838538:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019658.838541:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019658.838542:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019658.838546:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3921 00000100:00080000:3.0:1734019658.838548:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019658.838553:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019658.838557:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019658.838558:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333195904:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019658.838564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333196288/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019658.838573:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019658.838576:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019658.838578:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019658.838581:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3921 00000100:00080000:3.0:1734019658.838584:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019658.838587:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019658.838591:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019658.838598:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333196416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019658.838606:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019658.838609:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019658.838611:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019658.838614:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3921 00000100:00080000:3.0:1734019658.838617:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734019658.838621:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019658.838621:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019658.838625:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019658.838632:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333196544/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019658.838640:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734019658.838643:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019658/real 1734019658] req@ffff8800a73cea00 x1818247333195904/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019658.838644:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019658.838646:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019658.838649:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3921 00000100:00080000:3.0:1734019658.838652:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019658.838655:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019658.838659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019658.838663:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333195904/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019658.838667:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333196672/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019658.838673:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019658.838676:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019658.838681:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333195904:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019658.838688:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333196032:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019658.838706:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019658.838713:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019658/real 1734019658] req@ffff8800a73ce680 x1818247333196032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019658.838724:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333196032/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019658.838731:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019658.838735:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019658.838738:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333196032:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019658.838745:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333196160:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019658.838758:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019658.838764:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019658/real 1734019658] req@ffff8800a73cce00 x1818247333196160/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019658.838774:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333196160/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019658.838782:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019658.838785:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019658.838788:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333196160:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019658.838813:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333196288:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019658.838825:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019658.838872:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019658/real 1734019658] req@ffff8800a73ced80 x1818247333196288/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019658.838885:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333196288/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019658.838894:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019658.838896:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019658.838899:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333196288:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019658.838905:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333196416:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019658.838924:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019658.838930:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019658/real 1734019658] req@ffff8800a73cd880 x1818247333196416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019658.838941:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333196416/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019658.838949:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019658.838951:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019658.838954:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333196416:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019658.838959:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333196544:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019658.838971:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019658.838977:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019658/real 1734019658] req@ffff8800a73cca80 x1818247333196544/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019658.838986:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333196544/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019658.838993:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019658.838996:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019658.838999:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333196544:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019658.839004:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333196672:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019658.839016:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019658.839021:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019658/real 1734019658] req@ffff8800a73cd500 x1818247333196672/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019658.839038:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333196672/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019658.839046:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019658.839048:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019658.839051:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333196672:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019659.018469:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019660.020402:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019661.022441:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019662.024457:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019663.026450:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019663.846325:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019663.846334:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019663.846338:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019663.846348:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3926 00000100:00080000:3.0:1734019663.846357:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019663.846365:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019663.846400:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333196800/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019663.846434:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019663.846439:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019663.846440:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019663.846445:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3926 00000100:00080000:3.0:1734019663.846448:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019663.846452:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019663.846462:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019663.846473:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333196928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019663.846483:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019663.846486:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019663.846488:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019663.846491:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3926 00000100:00080000:3.0:1734019663.846494:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019663.846497:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019663.846502:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019663.846510:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333197056/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019663.846518:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019663.846521:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019663.846522:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019663.846526:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3926 00000100:00080000:3.0:1734019663.846528:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019663.846531:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019663.846536:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333196800:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734019663.846536:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019663.846543:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333197184/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019663.846551:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019663.846555:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019663.846557:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019663.846560:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3926 00000100:00080000:3.0:1734019663.846562:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019663.846566:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019663.846569:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019663.846577:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333197312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019663.846586:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019663.846588:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019663.846590:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019663.846594:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3926 00000100:00080000:3.0:1734019663.846596:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734019663.846600:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019663.846600:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019663.846604:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019663.846611:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333197440/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019663.846619:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734019663.846621:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019663/real 1734019663] req@ffff8800a73cf480 x1818247333196800/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019663.846622:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019663.846624:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019663.846627:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3926 00000100:00080000:3.0:1734019663.846630:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019663.846633:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019663.846637:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019663.846640:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333196800/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019663.846646:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333197568/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019663.846650:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019663.846653:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019663.846658:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333196800:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019663.846666:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333196928:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019663.846682:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019663.846689:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019663/real 1734019663] req@ffff8800a73cc380 x1818247333196928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019663.846700:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333196928/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019663.846708:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019663.846711:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019663.846714:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333196928:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019663.846721:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333197056:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019663.846734:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019663.846740:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019663/real 1734019663] req@ffff8800a73cf100 x1818247333197056/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019663.846801:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333197056/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019663.846810:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019663.846812:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019663.846816:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333197056:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019663.846839:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333197184:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019663.846856:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019663.846863:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019663/real 1734019663] req@ffff8800a73cc700 x1818247333197184/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019663.846874:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333197184/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019663.846881:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019663.846883:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019663.846886:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333197184:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019663.846891:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333197312:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019663.846907:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019663.846913:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019663/real 1734019663] req@ffff8800a73cdf80 x1818247333197312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019663.846923:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333197312/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019663.846931:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019663.846933:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019663.846936:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333197312:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019663.846941:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333197440:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019663.846952:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019663.846958:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019663/real 1734019663] req@ffff8800a73cf800 x1818247333197440/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019663.846968:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333197440/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019663.846975:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019663.846977:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019663.846980:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333197440:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019663.846985:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333197568:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019663.846997:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019663.847003:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019663/real 1734019663] req@ffff8800a73cfb80 x1818247333197568/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019663.847019:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333197568/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019663.847027:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019663.847029:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019663.847032:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333197568:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734019664.028443:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019665.030458:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019666.032352:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019667.034368:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019668.036405:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019668.854312:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019668.854318:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019668.854320:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019668.854327:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3931 00000100:00080000:3.0:1734019668.854333:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019668.854337:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019668.854359:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333197696/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019668.854379:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019668.854382:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019668.854383:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019668.854386:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3931 00000100:00080000:3.0:1734019668.854388:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019668.854390:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019668.854397:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019668.854406:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333197824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019668.854413:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019668.854415:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019668.854416:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019668.854418:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3931 00000100:00080000:3.0:1734019668.854421:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019668.854423:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019668.854425:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333197696:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734019668.854427:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019668.854432:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333197952/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019668.854438:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019668.854440:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019668.854441:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019668.854443:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3931 00000100:00080000:3.0:1734019668.854444:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019668.854446:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019668.854449:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019668.854454:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333198080/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019668.854459:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019668.854460:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019668.854461:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019668.854464:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3931 00000100:00080000:3.0:1734019668.854465:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1734019668.854467:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019668.854467:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019668.854470:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019668.854475:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333198208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019668.854482:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019668/real 1734019668] req@ffff8800a73cdc00 x1818247333197696/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019668.854484:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019668.854486:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019668.854487:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019668.854489:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3931 00000100:00080000:3.0:1734019668.854491:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1734019668.854492:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333197696/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019668.854493:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019668.854496:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1734019668.854499:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019668.854501:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:3.0:1734019668.854501:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333198336/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019668.854503:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333197696:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019668.854507:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734019668.854508:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333197824:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734019668.854509:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019668.854510:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019668.854512:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3931 00000100:00080000:3.0:1734019668.854514:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019668.854516:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734019668.854519:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734019668.854519:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019668.854523:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019668/real 1734019668] req@ffff8800a73cfb80 x1818247333197824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019668.854524:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333198464/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019668.854531:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333197824/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019668.854535:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019668.854537:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019668.854539:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333197824:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019668.854555:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333197952:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019668.854567:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019668.854572:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019668/real 1734019668] req@ffff8800a73cf800 x1818247333197952/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019668.854576:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333197952/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019668.854580:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019668.854581:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019668.854582:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333197952:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019668.854598:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333198080:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019668.854605:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019668.854607:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019668/real 1734019668] req@ffff8800a73cdf80 x1818247333198080/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019668.854612:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333198080/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019668.854615:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019668.854616:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019668.854617:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333198080:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019668.854619:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333198208:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019668.854626:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019668.854628:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019668/real 1734019668] req@ffff8800a73cc700 x1818247333198208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019668.854632:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333198208/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019668.854635:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019668.854636:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019668.854637:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333198208:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019668.854639:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333198336:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019668.854644:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019668.854647:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019668/real 1734019668] req@ffff8800a73cf100 x1818247333198336/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019668.854650:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333198336/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019668.854653:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019668.854654:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019668.854655:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333198336:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019668.854657:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333198464:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019668.854662:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019668.854691:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019668/real 1734019668] req@ffff8800a73cc380 x1818247333198464/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019668.854699:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333198464/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019668.854702:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019668.854703:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019668.854704:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333198464:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019669.038395:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019670.040471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019671.042361:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019672.044412:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019673.046542:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019673.862374:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019673.862383:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019673.862388:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019673.862398:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3936 00000100:00080000:3.0:1734019673.862407:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019673.862415:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019673.862451:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333198592/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019673.862484:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019673.862489:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019673.862490:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019673.862495:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3936 00000100:00080000:3.0:1734019673.862498:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019673.862502:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019673.862512:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019673.862522:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333198720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019673.862532:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019673.862536:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019673.862537:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019673.862541:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3936 00000100:00080000:3.0:1734019673.862544:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019673.862547:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019673.862552:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019673.862560:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333198848/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019673.862575:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019673.862578:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019673.862614:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019673.862618:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3936 00000100:00080000:3.0:1734019673.862621:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019673.862625:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019673.862629:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019673.862632:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333198592:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019673.862639:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333198976/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019673.862648:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019673.862651:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019673.862653:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019673.862656:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3936 00000100:00080000:3.0:1734019673.862659:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019673.862662:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019673.862666:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019673.862674:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333199104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019673.862682:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019673.862685:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019673.862687:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019673.862691:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3936 00000100:00080000:3.0:1734019673.862694:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019673.862697:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734019673.862698:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734019673.862701:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019673.862709:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333199232/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019673.862717:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1734019673.862719:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019673/real 1734019673] req@ffff8800a73cf480 x1818247333198592/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019673.862720:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019673.862721:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019673.862724:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3936 00000100:00080000:3.0:1734019673.862727:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019673.862730:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019673.862734:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019673.862738:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333198592/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019673.862743:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333199360/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:0.0:1734019673.862749:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019673.862753:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019673.862758:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333198592:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019673.862765:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333198720:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019673.862783:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019673.862789:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019673/real 1734019673] req@ffff8800a73cd500 x1818247333198720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019673.862800:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333198720/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019673.862808:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019673.862812:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019673.862815:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333198720:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019673.862821:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333198848:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019673.862835:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019673.862840:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019673/real 1734019673] req@ffff8800a73cca80 x1818247333198848/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019673.862850:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333198848/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019673.862858:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019673.862861:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019673.862864:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333198848:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019673.862888:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333198976:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019673.862900:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019673.862906:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019673/real 1734019673] req@ffff8800a73cd880 x1818247333198976/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019673.862916:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333198976/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019673.862924:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019673.862926:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019673.862929:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333198976:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019673.862934:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333199104:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019673.862949:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019673.862955:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019673/real 1734019673] req@ffff8800a73ced80 x1818247333199104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019673.862965:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333199104/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019673.862973:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019673.862975:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019673.862978:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333199104:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019673.862983:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333199232:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019673.862994:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019673.863000:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019673/real 1734019673] req@ffff8800a73cce00 x1818247333199232/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019673.863011:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333199232/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019673.863018:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019673.863020:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019673.863023:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333199232:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019673.863028:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333199360:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019673.863040:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019673.863045:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019673/real 1734019673] req@ffff8800a73ce680 x1818247333199360/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019673.863062:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333199360/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019673.863070:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019673.863072:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019673.863075:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333199360:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734019674.048396:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019675.050457:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019676.052443:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019677.054351:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019678.056365:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019678.870336:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019678.870343:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019678.870346:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019678.870354:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3941 00000100:00080000:3.0:1734019678.870361:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019678.870366:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019678.870392:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333199488/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019678.870419:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019678.870422:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019678.870423:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019678.870425:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3941 00000100:00080000:3.0:1734019678.870427:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019678.870429:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019678.870436:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019678.870444:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333199616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019678.870449:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019678.870451:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019678.870451:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019678.870453:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3941 00000100:00080000:3.0:1734019678.870455:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019678.870456:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019678.870459:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019678.870463:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333199744/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019678.870467:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019678.870469:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019678.870470:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019678.870472:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3941 00000100:00080000:3.0:1734019678.870473:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019678.870475:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019678.870477:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019678.870481:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333199872/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019678.870485:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019678.870487:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019678.870488:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019678.870489:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3941 00000100:00080000:3.0:1734019678.870490:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019678.870492:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:0.0:1734019678.870514:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333199488:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734019678.870520:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019678.870526:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333200000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019678.870531:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019678.870532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019678.870533:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019678.870535:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3941 00000100:00080000:3.0:1734019678.870537:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019678.870538:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019678.870541:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019678.870545:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333200128/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019678.870549:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019678.870551:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019678.870552:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019678.870554:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3941 00000100:00080000:3.0:1734019678.870555:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019678.870557:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000800:00000100:0.0:1734019678.870559:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000008:00100000:3.0:1734019678.870560:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019678.870564:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333200256/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019678.870574:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019678/real 1734019678] req@ffff8800a73cea00 x1818247333199488/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019678.870585:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333199488/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019678.870590:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019678.870592:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019678.870595:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333199488:192.168.201.152@tcp:250:kworker.0 00000100:00100000:0.0:1734019678.870599:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333199616:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019678.870609:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019678.870612:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019678/real 1734019678] req@ffff8800a73ce680 x1818247333199616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019678.870617:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333199616/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019678.870621:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019678.870624:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019678.870625:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333199616:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019678.870629:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333199744:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019678.870636:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019678.870639:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019678/real 1734019678] req@ffff8800a73cce00 x1818247333199744/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019678.870644:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333199744/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019678.870648:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019678.870650:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019678.870651:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333199744:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019678.870654:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333199872:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019678.870660:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019678.870663:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019678/real 1734019678] req@ffff8800a73ced80 x1818247333199872/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019678.870668:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333199872/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019678.870671:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019678.870673:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019678.870675:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333199872:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019678.870692:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333200000:192.168.201.152@tcp:38:kworker.0 00000800:00000100:0.0:1734019678.870700:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019678.870704:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019678/real 1734019678] req@ffff8800a73cd880 x1818247333200000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019678.870709:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333200000/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019678.870713:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019678.870714:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019678.870716:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333200000:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019678.870718:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333200128:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019678.870724:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019678.870727:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019678/real 1734019678] req@ffff8800a73cca80 x1818247333200128/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019678.870732:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333200128/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019678.870735:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019678.870737:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019678.870738:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333200128:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019678.870742:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333200256:192.168.201.152@tcp:8:kworker.0 00000800:00000100:0.0:1734019678.870748:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:0.0:1734019678.870750:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019678/real 1734019678] req@ffff8800a73cd500 x1818247333200256/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:0.0:1734019678.870759:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333200256/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:0.0:1734019678.870764:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1734019678.870765:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:0.0:1734019678.870766:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333200256:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019679.058354:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019680.060424:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019681.062367:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019682.064530:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019683.066514:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019683.878343:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019683.878351:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019683.878356:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019683.878366:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3946 00000100:00080000:3.0:1734019683.878376:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019683.878383:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019683.878452:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333200384/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019683.878485:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019683.878489:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019683.878491:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019683.878495:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3946 00000100:00080000:3.0:1734019683.878499:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019683.878503:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019683.878513:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019683.878524:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333200512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019683.878535:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019683.878538:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019683.878540:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019683.878544:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3946 00000100:00080000:3.0:1734019683.878546:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019683.878549:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019683.878554:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019683.878562:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333200640/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019683.878571:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019683.878574:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019683.878575:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019683.878579:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3946 00000100:00080000:3.0:1734019683.878581:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019683.878584:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019683.878589:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1734019683.878595:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333200384:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019683.878596:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333200768/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019683.878605:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019683.878608:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019683.878610:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019683.878614:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3946 00000100:00080000:3.0:1734019683.878617:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019683.878620:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019683.878625:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019683.878632:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333200896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019683.878641:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019683.878644:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019683.878645:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019683.878649:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3946 00000100:00080000:3.0:1734019683.878652:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019683.878655:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019683.878659:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019683.878666:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333201024/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019683.878674:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019683.878677:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019683.878679:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019683.878682:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3946 00000100:00100000:0.0:1734019683.878685:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333200512:192.168.201.152@tcp:38:kworker.0 00000100:00080000:3.0:1734019683.878685:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019683.878689:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019683.878693:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019683.878702:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333201152/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:0.0:1734019683.878705:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333200640:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019683.878728:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333200768:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019683.878742:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333200896:192.168.201.152@tcp:38:kworker.0 00000100:00100000:0.0:1734019683.878756:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333201024:192.168.201.152@tcp:8:kworker.0 00000100:00100000:0.0:1734019683.878770:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333201152:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734019684.068498:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019685.070483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019686.072438:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1734019686.884503:0:1958:0:(lib-socket.c:428:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.152:988 00000400:00000100:1.0:1734019686.884539:0:1958:0:(acceptor.c:84:lnet_connect_console_error()) Connection to 192.168.201.152@tcp at host 192.168.201.152 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1734019686.884553:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019686.884577:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019686.884582:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019686.884586:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019686.884589:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019686.884593:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000800:00000100:1.0:1734019686.884597:0:1958:0:(socklnd_cb.c:423:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.152@tcp 00000100:00100000:2.0:1734019686.884652:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019683/real 1734019686] req@ffff8800a73cc700 x1818247333200768/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019686.884669:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333200768/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019686.884676:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019686.884678:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019686.884681:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333200768:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019686.884685:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019683/real 1734019686] req@ffff8800a73cdf80 x1818247333200896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019686.884691:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333200896/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019686.884695:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019686.884697:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019686.884700:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333200896:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019686.884703:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019683/real 1734019686] req@ffff8800a73cf800 x1818247333201024/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019686.884709:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333201024/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019686.884713:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019686.884714:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019686.884716:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333201024:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019686.884719:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019683/real 1734019686] req@ffff8800a73cfb80 x1818247333201152/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019686.884724:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333201152/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019686.884728:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019686.884729:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019686.884731:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333201152:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019686.884734:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019683/real 1734019686] req@ffff8800a73cf480 x1818247333200384/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019686.884740:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333200384/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019686.884743:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019686.884745:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019686.884746:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333200384:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019686.884750:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019683/real 1734019686] req@ffff8800a73cc380 x1818247333200512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019686.884756:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333200512/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019686.884759:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019686.884761:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019686.884762:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333200512:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019686.884765:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019683/real 1734019686] req@ffff8800a73cf100 x1818247333200640/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019686.884771:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333200640/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019686.884775:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019686.884776:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019686.884777:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333200640:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019687.074435:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019688.076440:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019688.886317:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019688.886325:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019688.886351:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019688.886359:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3951 00000100:00080000:3.0:1734019688.886367:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019688.886373:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019688.886398:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333201280/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019688.886420:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019688.886422:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019688.886423:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019688.886426:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3951 00000100:00080000:3.0:1734019688.886428:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019688.886430:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019688.886437:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019688.886445:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333201408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019688.886450:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019688.886452:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019688.886453:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019688.886455:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3951 00000100:00080000:3.0:1734019688.886457:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019688.886459:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019688.886462:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019688.886466:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333201536/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019688.886470:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019688.886472:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019688.886473:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019688.886475:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3951 00000100:00080000:3.0:1734019688.886476:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019688.886478:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019688.886480:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019688.886484:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333201664/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019688.886488:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019688.886489:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019688.886490:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019688.886492:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3951 00000100:00080000:3.0:1734019688.886493:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019688.886495:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019688.886497:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019688.886501:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333201792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019688.886502:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333201280:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019688.886509:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019688.886511:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019688.886512:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019688.886514:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3951 00000100:00080000:3.0:1734019688.886515:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019688.886517:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019688.886519:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019688.886523:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333201920/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019688.886527:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019688.886528:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019688.886529:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019688.886531:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3951 00000100:00080000:3.0:1734019688.886539:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019688.886541:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019688.886544:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019688.886546:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019688.886548:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333202048/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019688.886560:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019688/real 1734019688] req@ffff8800a73cdc00 x1818247333201280/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019688.886570:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333201280/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019688.886576:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019688.886578:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019688.886580:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333201280:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019688.886584:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333201408:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019688.886593:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019688.886596:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019688/real 1734019688] req@ffff8800a73cf100 x1818247333201408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019688.886602:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333201408/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019688.886606:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019688.886608:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019688.886609:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333201408:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019688.886612:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333201536:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019688.886619:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019688.886622:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019688/real 1734019688] req@ffff8800a73cc380 x1818247333201536/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019688.886627:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333201536/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019688.886631:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019688.886632:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019688.886634:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333201536:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019688.886637:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333201664:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019688.886645:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019688.886648:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019688/real 1734019688] req@ffff8800a73cf480 x1818247333201664/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019688.886653:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333201664/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019688.886657:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019688.886658:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019688.886659:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333201664:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019688.886674:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333201792:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019688.886680:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019688.886683:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019688/real 1734019688] req@ffff8800a73cfb80 x1818247333201792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019688.886689:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333201792/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019688.886693:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019688.886694:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019688.886695:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333201792:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019688.886698:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333201920:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019688.886704:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019688.886707:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019688/real 1734019688] req@ffff8800a73cf800 x1818247333201920/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019688.886715:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333201920/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019688.886719:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019688.886720:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019688.886722:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333201920:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019688.886725:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333202048:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019688.886730:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019688.886733:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019688/real 1734019688] req@ffff8800a73cdf80 x1818247333202048/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019688.886738:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333202048/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019688.886741:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019688.886743:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019688.886744:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333202048:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019689.078436:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019690.080418:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019691.082441:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019692.084457:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019693.086485:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019693.894353:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019693.894362:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019693.894367:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019693.894377:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3956 00000100:00080000:3.0:1734019693.894386:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019693.894394:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019693.894430:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333202176/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019693.894463:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019693.894468:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019693.894469:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019693.894474:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3956 00000100:00080000:3.0:1734019693.894477:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019693.894481:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019693.894491:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019693.894501:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333202304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019693.894511:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019693.894514:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019693.894516:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019693.894519:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3956 00000100:00080000:3.0:1734019693.894522:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019693.894526:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019693.894530:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019693.894538:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333202432/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019693.894554:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019693.894557:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019693.894559:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019693.894562:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3956 00000100:00080000:3.0:1734019693.894565:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019693.894568:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000100:00100000:2.0:1734019693.894571:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333202176:192.168.201.152@tcp:250:kworker.0 00000008:00100000:3.0:1734019693.894573:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019693.894580:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333202560/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019693.894589:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019693.894592:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019693.894594:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019693.894597:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3956 00000100:00080000:3.0:1734019693.894600:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019693.894603:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019693.894607:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019693.894614:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333202688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019693.894623:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019693.894626:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019693.894627:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019693.894631:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3956 00000100:00080000:3.0:1734019693.894634:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734019693.894635:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019693.894638:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019693.894642:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019693.894649:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333202816/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019693.894657:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019693/real 1734019693] req@ffff8800a73cc700 x1818247333202176/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019693.894657:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019693.894660:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019693.894662:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019693.894665:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3956 00000100:00080000:3.0:1734019693.894668:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019693.894671:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019693.894675:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019693.894676:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333202176/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019693.894685:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333202944/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019693.894688:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019693.894691:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019693.894696:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333202176:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019693.894703:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333202304:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019693.894719:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019693.894726:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019693/real 1734019693] req@ffff8800a73cd500 x1818247333202304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019693.894737:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333202304/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019693.894745:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019693.894747:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019693.894751:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333202304:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019693.894756:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333202432:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019693.894769:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019693.894775:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019693/real 1734019693] req@ffff8800a73cca80 x1818247333202432/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019693.894785:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333202432/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019693.894793:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019693.894795:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019693.894798:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333202432:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019693.894822:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333202560:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019693.894834:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019693.894840:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019693/real 1734019693] req@ffff8800a73cd880 x1818247333202560/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019693.894851:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333202560/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019693.894858:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019693.894860:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019693.894863:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333202560:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019693.894868:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333202688:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019693.894880:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019693.894886:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019693/real 1734019693] req@ffff8800a73ced80 x1818247333202688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019693.894895:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333202688/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019693.894903:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019693.894905:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019693.894908:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333202688:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019693.894913:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333202816:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019693.894925:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019693.894931:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019693/real 1734019693] req@ffff8800a73cce00 x1818247333202816/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019693.894942:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333202816/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019693.894949:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019693.894951:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019693.894954:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333202816:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019693.894960:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333202944:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019693.894971:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019693.894977:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019693/real 1734019693] req@ffff8800a73ce680 x1818247333202944/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019693.894987:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333202944/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019693.894994:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019693.894996:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019693.894999:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333202944:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734019694.088487:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019695.090452:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019696.092446:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019697.094444:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019698.096462:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019698.902354:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019698.902363:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019698.902368:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019698.902378:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3961 00000100:00080000:3.0:1734019698.902394:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019698.902402:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019698.902438:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cea00 x1818247333203072/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019698.902479:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019698.902483:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019698.902485:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019698.902490:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3961 00000100:00080000:3.0:1734019698.902493:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019698.902497:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019698.902507:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019698.902519:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ce680 x1818247333203200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019698.902528:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019698.902532:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019698.902533:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019698.902537:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3961 00000100:00080000:3.0:1734019698.902540:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019698.902543:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019698.902548:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019698.902557:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cce00 x1818247333203328/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019698.902565:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019698.902568:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019698.902570:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019698.902573:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3961 00000100:00080000:3.0:1734019698.902576:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019698.902579:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019698.902584:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019698.902589:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333203072:192.168.201.152@tcp:250:kworker.0 00000100:00100000:3.0:1734019698.902592:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73ced80 x1818247333203456/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019698.902601:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019698.902604:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019698.902605:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019698.902609:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3961 00000100:00080000:3.0:1734019698.902611:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019698.902614:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019698.902618:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019698.902626:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd880 x1818247333203584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019698.902634:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019698.902637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019698.902639:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019698.902642:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3961 00000100:00080000:3.0:1734019698.902645:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019698.902648:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019698.902653:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019698.902656:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019698.902661:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cca80 x1818247333203712/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019698.902669:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019698.902673:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019698.902675:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019698.902676:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019698/real 1734019698] req@ffff8800a73cea00 x1818247333203072/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019698.902678:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3961 00000100:00080000:3.0:1734019698.902681:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019698.902684:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019698.902689:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019698.902695:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cea00 x1818247333203072/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019698.902697:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cd500 x1818247333203840/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019698.902707:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019698.902710:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019698.902715:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333203072:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019698.902722:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333203200:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019698.902739:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019698.902745:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019698/real 1734019698] req@ffff8800a73ce680 x1818247333203200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019698.902756:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ce680 x1818247333203200/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019698.902764:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019698.902767:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019698.902770:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333203200:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019698.902776:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333203328:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019698.902789:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019698.902795:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019698/real 1734019698] req@ffff8800a73cce00 x1818247333203328/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019698.902805:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cce00 x1818247333203328/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019698.902812:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019698.902815:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019698.902818:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333203328:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019698.902843:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333203456:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019698.902856:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019698.902862:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019698/real 1734019698] req@ffff8800a73ced80 x1818247333203456/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019698.902873:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73ced80 x1818247333203456/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019698.902881:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019698.902883:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019698.902886:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333203456:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019698.902891:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333203584:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019698.902902:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019698.902908:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019698/real 1734019698] req@ffff8800a73cd880 x1818247333203584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019698.902917:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd880 x1818247333203584/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019698.902925:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019698.902928:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019698.902931:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333203584:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019698.902936:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333203712:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019698.902947:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019698.902953:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019698/real 1734019698] req@ffff8800a73cca80 x1818247333203712/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019698.902964:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cca80 x1818247333203712/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019698.902972:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019698.902974:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019698.902977:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333203712:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019698.902982:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333203840:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019698.902993:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019698.902999:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019698/real 1734019698] req@ffff8800a73cd500 x1818247333203840/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019698.903009:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cd500 x1818247333203840/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019698.903016:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019698.903018:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019698.903021:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333203840:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019699.098491:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019700.100471:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019701.102472:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019702.104496:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019703.106482:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019703.910370:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019703.910378:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019703.910382:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019703.910393:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3966 00000100:00080000:3.0:1734019703.910402:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019703.910410:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019703.910444:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc700 x1818247333203968/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019703.910477:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019703.910481:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019703.910483:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019703.910488:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3966 00000100:00080000:3.0:1734019703.910491:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019703.910495:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019703.910505:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019703.910516:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333204096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019703.910526:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019703.910530:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019703.910532:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019703.910535:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3966 00000100:00080000:3.0:1734019703.910537:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019703.910541:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019703.910546:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019703.910553:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333204224/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019703.910562:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019703.910565:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019703.910567:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019703.910570:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3966 00000100:00080000:3.0:1734019703.910573:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1734019703.910576:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333203968:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019703.910576:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019703.910580:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019703.910588:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333204352/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019703.910596:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019703.910599:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019703.910601:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019703.910605:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3966 00000100:00080000:3.0:1734019703.910607:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019703.910611:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019703.910615:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019703.910622:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333204480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019703.910630:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019703.910633:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019703.910635:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019703.910638:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3966 00000100:00080000:3.0:1734019703.910641:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1734019703.910643:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00080000:3.0:1734019703.910645:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019703.910649:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019703.910656:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333204608/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019703.910664:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019703/real 1734019703] req@ffff8800a73cc700 x1818247333203968/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:3.0:1734019703.910664:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019703.910667:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019703.910668:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019703.910672:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3966 00000100:00080000:3.0:1734019703.910674:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019703.910677:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019703.910682:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1734019703.910683:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc700 x1818247333203968/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:3.0:1734019703.910691:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333204736/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:2.0:1734019703.910694:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019703.910697:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019703.910702:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333203968:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019703.910709:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333204096:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019703.910726:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019703.910733:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019703/real 1734019703] req@ffff8800a73cdf80 x1818247333204096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019703.910744:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333204096/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019703.910752:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019703.910755:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019703.910758:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333204096:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019703.910780:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333204224:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019703.910794:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019703.910800:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019703/real 1734019703] req@ffff8800a73cf800 x1818247333204224/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019703.910811:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333204224/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019703.910819:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019703.910821:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019703.910824:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333204224:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019703.910830:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333204352:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019703.910842:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019703.910848:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019703/real 1734019703] req@ffff8800a73cfb80 x1818247333204352/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019703.910858:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333204352/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019703.910865:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019703.910868:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019703.910871:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333204352:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019703.910876:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333204480:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019703.910887:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019703.910893:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019703/real 1734019703] req@ffff8800a73cf480 x1818247333204480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019703.910903:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333204480/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019703.910910:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019703.910912:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019703.910915:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333204480:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019703.910920:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333204608:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019703.910932:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019703.910938:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019703/real 1734019703] req@ffff8800a73cc380 x1818247333204608/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019703.910948:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333204608/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019703.910956:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019703.910958:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019703.910961:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333204608:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019703.910966:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333204736:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019703.910977:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019703.910983:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019703/real 1734019703] req@ffff8800a73cf100 x1818247333204736/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019703.910992:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333204736/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019703.910999:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019703.911001:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019703.911004:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333204736:192.168.201.152@tcp:8:kworker.0 00010000:00010000:2.0:1734019704.108430:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019705.110483:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019706.112481:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1734019707.114454:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1734019708.116475:0:5236:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1734019708.918363:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 356ef100-c450-4ad1-ae36-19e8c4618593->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019708.918372:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1734019708.918376:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019708.918386:0:5:0:(import.c:523:import_select_connection()) MGC192.168.201.152@tcp: connect to NID 192.168.201.152@tcp last attempt 3971 00000100:00080000:3.0:1734019708.918395:0:5:0:(import.c:580:import_select_connection()) MGC192.168.201.152@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019708.918403:0:5:0:(import.c:615:import_select_connection()) MGC192.168.201.152@tcp: import ffff8800b58a5800 using connection MGC192.168.201.152@tcp_0/192.168.201.152@tcp 00000100:00100000:3.0:1734019708.918437:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdc00 x1818247333204864/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019708.918472:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019708.918476:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019708.918478:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019708.918483:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3971 00000100:00080000:3.0:1734019708.918486:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019708.918490:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b5805800: import ffff8800a9471000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019708.918499:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019708.918511:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf100 x1818247333204992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019708.918521:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019708.918524:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019708.918526:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019708.918529:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3971 00000100:00080000:3.0:1734019708.918534:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019708.918537:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b5805800: import ffff8800aa6a0800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019708.918542:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 32768, lost: 0. 00000100:00100000:3.0:1734019708.918550:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cc380 x1818247333205120/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019708.918559:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) 5762f609-0e87-4e8c-9bd6-d528092c9d9c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019708.918562:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019708.918564:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019708.918567:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: connect to NID 192.168.201.152@tcp last attempt 3971 00000100:00080000:3.0:1734019708.918569:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019708.918572:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b5805800: import ffff8800aa6a2000 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019708.918577:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019708.918584:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf480 x1818247333205248/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019708.918593:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019708.918596:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1734019708.918598:0:5:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1734019708.918601:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333204864:192.168.201.152@tcp:250:kworker.0 00000100:00080000:3.0:1734019708.918602:0:5:0:(import.c:523:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3971 00000100:00080000:3.0:1734019708.918604:0:5:0:(import.c:580:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019708.918608:0:5:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff8800b3cc4800: import ffff8800aa6a5800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019708.918612:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34603008 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1734019708.918620:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cfb80 x1818247333205376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019708.918634:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019708.918637:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1734019708.918639:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019708.918643:0:5:0:(import.c:523:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3971 00000100:00080000:3.0:1734019708.918646:0:5:0:(import.c:580:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019708.918649:0:5:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff8800b3cc4800: import ffff88012a632800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019708.918653:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1734019708.918660:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cf800 x1818247333205504/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00080000:3.0:1734019708.918668:0:5:0:(pinger.c:254:ptlrpc_pinger_process_import()) f1fa65c5-ac21-42fc-9fc8-6a110334833e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1734019708.918671:0:5:0:(pinger.c:262:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1734019708.918673:0:5:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1734019708.918676:0:5:0:(import.c:523:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: connect to NID 192.168.201.152@tcp last attempt 3971 00000100:00080000:3.0:1734019708.918679:0:5:0:(import.c:580:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1734019708.918682:0:5:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff8800b3cc4800: import ffff88012a637800 using connection 192.168.201.152@tcp/192.168.201.152@tcp 00000008:00100000:3.0:1734019708.918686:0:5:0:(osc_request.c:3668:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34603008 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1734019708.918688:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:3.0:1734019708.918693:0:5:0:(import.c:814:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8800a73cdf80 x1818247333205632/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 00000100:00100000:2.0:1734019708.918717:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019708/real 1734019708] req@ffff8800a73cdc00 x1818247333204864/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019708.918738:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdc00 x1818247333204864/t0(0) o250->MGC192.168.201.152@tcp@192.168.201.152@tcp:26/25 lens 520/544 e 0 to 1 dl 1734019733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019708.918752:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800b58a5800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019708.918756:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019708.918762:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:356ef100-c450-4ad1-ae36-19e8c4618593:1967:1818247333204864:192.168.201.152@tcp:250:kworker.0 00000100:00100000:2.0:1734019708.918770:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333204992:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019708.918788:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019708.918794:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019708/real 1734019708] req@ffff8800a73cf100 x1818247333204992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019708.918805:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf100 x1818247333204992/t0(0) o38->lustre-MDT0000-mdc-ffff8800b5805800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019708.918813:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800a9471000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019708.918816:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019708.918820:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333204992:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019708.918827:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333205120:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019708.918842:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019708.918847:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019708/real 1734019708] req@ffff8800a73cc380 x1818247333205120/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019708.918858:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cc380 x1818247333205120/t0(0) o8->lustre-OST0000-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019708.918866:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a0800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019708.918869:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019708.918872:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333205120:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019708.918904:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333205248:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019708.918918:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019708.918924:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019708/real 1734019708] req@ffff8800a73cf480 x1818247333205248/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019708.918935:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf480 x1818247333205248/t0(0) o8->lustre-OST0001-osc-ffff8800b5805800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019708.918943:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a2000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019708.918945:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019708.918948:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5762f609-0e87-4e8c-9bd6-d528092c9d9c:1967:1818247333205248:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019708.918954:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333205376:192.168.201.152@tcp:38:kworker.0 00000800:00000100:2.0:1734019708.918967:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019708.918973:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019708/real 1734019708] req@ffff8800a73cfb80 x1818247333205376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019708.918982:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cfb80 x1818247333205376/t0(0) o38->lustre-MDT0000-mdc-ffff8800b3cc4800@192.168.201.152@tcp:12/10 lens 520/544 e 0 to 1 dl 1734019733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019708.919039:0:1967:0:(import.c:86:import_set_state_nolock()) ffff8800aa6a5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019708.919043:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019708.919047:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333205376:192.168.201.152@tcp:38:kworker.0 00000100:00100000:2.0:1734019708.919053:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333205504:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019708.919071:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019708.919079:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019708/real 1734019708] req@ffff8800a73cf800 x1818247333205504/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019708.919091:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cf800 x1818247333205504/t0(0) o8->lustre-OST0000-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019708.919099:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a632800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019708.919101:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019708.919104:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333205504:192.168.201.152@tcp:8:kworker.0 00000100:00100000:2.0:1734019708.919110:0:1967:0:(client.c:1796:ptlrpc_send_new_req()) Sending RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333205632:192.168.201.152@tcp:8:kworker.0 00000800:00000100:2.0:1734019708.919123:0:1967:0:(socklnd_cb.c:975:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.152@tcp 00000100:00100000:2.0:1734019708.919128:0:1967:0:(client.c:2363:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1734019708/real 1734019708] req@ffff8800a73cdf80 x1818247333205632/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00100000:2.0:1734019708.919138:0:1967:0:(client.c:2394:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8800a73cdf80 x1818247333205632/t0(0) o8->lustre-OST0001-osc-ffff8800b3cc4800@192.168.201.152@tcp:28/4 lens 520/544 e 0 to 1 dl 1734019733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 00000100:00080000:2.0:1734019708.919145:0:1967:0:(import.c:86:import_set_state_nolock()) ffff88012a637800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1734019708.919148:0:1967:0:(import.c:1479:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.152@tcp failed (-110) 00000100:00100000:2.0:1734019708.919151:0:1967:0:(client.c:2281:ptlrpc_check_set()) Completed RPC req@ffff8800a73cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f1fa65c5-ac21-42fc-9fc8-6a110334833e:1967:1818247333205632:192.168.201.152@tcp:8:kworker.0 00010000:00010000:3.0:1734019709.118501:0:4737:0:(ldlm_lockd.c:2841:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 75547 lines, 75547 kept, 0 dropped, 0 bad.